Trying to use ExPVR with German Sky+

Post here if you have any problems with ExPVR.
VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 8:17 am

Hello,

I'm a subscriber to the German Sky service. I am not entirely sure how close they are connected to the UK Sky service technology wise but it seems quite close. Services such as "Sky+" and "Anytime" are available and from what I read, they seem to be similar to what is offered in the UK. The Hardware available in Germany looks different from what is shown on the UK website (although the remotes are pretty similar). The Menu system looks different too but the features and service names are pretty much the same so I assume there is quite a big technology overlap.

Recently, an upgrade from the Sky provided 320GB external harddrives for Sky+/Anytime to 2TB drives was made available and I was looking to transfer recordings from the 320GB drive to the new 2TB drive. I was able to access the Drive and it looks like the System is based on XTV (S<nnn> directories with recorded streams and metadata). However ExPVR was not able to recognize the drive. It displayed a progress with "scanning recordings" and then showed an error along the lines of "no database found, are you sure this is a Sky+ disk) - I also found a press release on the NDS site stating that Sky-Germany uses XTV http://www.nds.com/WorkArea/DownloadAsset.aspx?id=414

From what I could find out about Sky/XTV, there should be an FSN_DATA directory on the drive. This is not present in my case. Just the S<nnn> dirs, the ROOT_DIR.CP0 and OPENEDFI.LES - I'm not sure about the role of the FSN_DATA dir but it seems to contain what is called the "planner database". Again I don't know yet what the planner database is exactly. Assuming it has information about planned recordings, it seems like Sky Germany keeps that information directly in the box (flash memory?) and not on the Harddrive because scheduled recordings are still scheduled after the external harddrives are swopped.

Do you know what would happen if I just transferred the S<nnn> directories to the new drive? Would the recordings be found by the Receiver. Has the ROOT_DIR.CP0 file to be updated when things are added to the drive? I understand it's a copy of the FAT root directory data, so I might be able to do that manually if it is required.

Might there be some other things I could try in order to use ExPVR to transfer the recordings?

Let me know if I should provide more information... I know that a lot of Sky subscribers in Germany are facing similar problems with the rollout of the larger Harddrive...

Bests,
Lars

mrmt32
Site Admin
Posts: 585
Joined: Mon Jul 25, 2011 1:05 am

Re: Trying to use ExPVR with German Sky+

Postby mrmt32 » Fri Apr 12, 2013 10:27 am

That is definitely an XTV disk, and it probably equivalent to the UK's post-Darwin format (from looking at the Sxxx folders).

Now the planner database does indeed keep track of recordings. It is not just planned recordings though, but also previously recorded ones. There is also another database (MFS) which contains some data which is required to read some files (any files that were recorded back to back on the same channel really need this file to be extracted).

Copying the files across will unfortunately not work. Windows explorer is not able to deal with the XTVFS format, the reason why you can see some files is because XTVFS is highly based on FAT32 (which windows can read). If you tried to copy using windows explorer, the stream.str files (that contain the video data) would not copy.

Now, a while ago (back when ExPVR was relatively new), someone from Germany did try and use ExPVR (I think this was with an internal disk though). It turned out the disk had two partitions; the normal XTVFS one which contained the recordings and an EXT2 (this is a Linux filesystem) partition containing the FSN_DATA files. Would you be able to take a look at the drive in disk manager and see if the partition is present?

I am not sure how the external disk system will work with your box. We do not have support for external disks in the UK (although, interestingly, the eSATA port has recently become active, the plugged in external disk just overrides the internal one however and this has never been publicised by sky). Is the external drive the only disk your boxes have?

If your disk is the same (ext2 partition), then adding support I'm ExPVR is going to be tricky as it would mean implementing another filesystem. Saying that, getting support for German boxes would be very good, so if this is the case I will try and find a way of adding in support.

Edit: the post from before about German planners is at: viewtopic.php?f=6&t=35

VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Re: Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 11:26 am

Thanks for the response!

Yes, there is indeed another (smaller) partition. I just had a quick look last night, don't know which type of filesystem it is yet. I will check again and report back.

Sky Germany is using mostly external Hard Disks today in conjunctions with receivers built by Humax or Pace. There is also a model from Pace that has an internal HDD but that seems to be rather uncommon these days. The external hard drives come in a special enclosure that is placed underneath the actual receiver and connected with a USB+eSATA Y-cable. The drives are not sold but rented out to the subscribers. They are sealed so it's problematic to open them up in order to get to the hard drive. To make things more interesting, the drives standby-status seems to be controlled via the USB connection so just hooking the eSATA and USB up to a computer won't operate the drive. However, I found that keeping the USB connection hooked up to the receiver and connecting the eSATA part of the Y-Cable to the computer will keep the drive powered up so that the eSATA connection to the computer works. This is how the German 2TB setup looks like:
Image

Here's a bit more detail about what I saw last night (first had it hooked up to a mac, so the output is in unix/shell style):
root dir of the XTVFS partition:
drwxrwxrwx 1 100 100 32768 Dec 31 1999 OPENEDFI.LES
-rwxrwxrwx 1 100 100 32768 Dec 31 1999 ROOT_DIR.CP0
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S1010
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S1011
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S1012
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S1014
...
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S254
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S401
drwxrwxrwx 1 100 100 32768 Jan 1 2000 S418

cd S254
total 1187776
-rwxrwxrwx 1 100 100 45936 Jan 1 2000 00000001.XMA
-rwxrwxrwx 1 100 100 102000 Jan 1 2000 00000001.XMI
-rwxrwxrwx 1 100 100 143620 Jan 1 2000 00000001.XMV
-rwxrwxrwx 1 100 100 4276 Jan 1 2000 00000002.XMA
-rwxrwxrwx 1 100 100 8888 Jan 1 2000 00000002.XMI
-rwxrwxrwx 1 100 100 12795 Jan 1 2000 00000002.XMV
-rwxrwxrwx 1 100 100 64 Jan 1 2000 META_MAN.MTA
-rwxrwxrwx 1 100 100 72 Jan 1 2000 META_MAN.MTI
-rwxrwxrwx 1 100 100 971 Jan 1 2000 META_MAN.XMD
-rwxrwxrwx 1 100 100 5400 Jan 1 2000 STREAM.EXN
-rwxrwxrwx 1 100 100 607519232 Jan 1 2000 STREAM.STR

I think the other partition was rather small, so I guess I can create an image of that one, compress it and upload it if you care to take a look yourself...

Thanks...
Lars

mrmt32
Site Admin
Posts: 585
Joined: Mon Jul 25, 2011 1:05 am

Re: Trying to use ExPVR with German Sky+

Postby mrmt32 » Fri Apr 12, 2013 11:46 am

That is really interesting, I had no idea that some platforms had the hard drive external to the box. Does the hard drive have its own power connector, or does it only have USB and eSATA connections? If it is only those, then the power is probably provided through USB (normal eSATA does not carry power). Maybe the sky box is able to provide a greater current than a PC (USB is not normally enough to power a 3.5" disk, which I presume is inside the box).

That layout is exactly the same as UK boxes, except for the lack of the FSN_DATA directory. It does seem highly likely that the other partition is EXT2 with the databases on it. If you could send me an image of it, that would be great. If you cannot upload it to dropbox or similar because it is too big, I can give you PM some FTP details. If it does just contain the databases, it should compress down well though; the files on there will probably only go to a maximum of about 100MB and most of this is the EPG cache which should be highly compressible.

I would be careful connecting the disk to a mac computer. I only say this because there have been cases before where mac computers have automatically tried to 'fix' the drive, thinking it was supposed to be standard FAT32. The may only happen if you boot the computer with the disk connected, but I have never tested this out. In normal cases, the damage is fixable by running it through ExPVR, but it could be a bit more tricky with your disk due to the current incompatibility.

VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Re: Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 12:21 pm

The older 320GB disk is powered through the USB port of the box. The USB port is specifically market as providing 1.0A of current. However, hooking up the USB part of the Y-Cable to a computer shows an USB input devices (PACE 100 -something- HDD) and given the standby behavior associated to the USB connection (red LED/green LED) there seems to be more going on. The 320GB enclosure is much flatter then the 2TB one. I'm certain it's a 2.5" drive in there. Here's an image of that one:
Image

The 2TB enclosure comes with another Y-adapter that splits up the receiver's DC power connection to also go to the drive enclosure. however, it also uses the USB/eSATA Y-Cable. The enclosure might even host two 1TB drives... I have not hooked it up to a computer yet. Since they are forcing a 1:1 devision of the space between individual recordings and Anytime content, that would be possible. I'll check that out, and I'll be careful about connecting it to the Mac (which also wants to create some hidden files all the time and given that this is not really FAT, there is this chance of corrupting stream data)

mrmt32
Site Admin
Posts: 585
Joined: Mon Jul 25, 2011 1:05 am

Re: Trying to use ExPVR with German Sky+

Postby mrmt32 » Fri Apr 12, 2013 5:43 pm

Interesting, I have never seen anything like that before. The sky box must be sending command through the USB to turn it on then, how this works would probably be quite hard to find out though without some kind of USB sniffing equipment... If the device is physically smaller, then you are probably right about the 2.5" disk; this would also be able to be powered by USB.

I would expect that the 2TB disk would contain a single 2TB disk, two 1TB 2.5" drives would be far more costly and would require significant software changes. The anytime recordings are stored on the same partition, there is no physical separation of the data. This makes it easy for the box to move recordings from anytime to your planner (it just has to mark the recording as "user" instead of "system" in the planner database).

One other thing that would be useful is if you could send me the ExPVR log from when you loaded the disk. This will contain some details about where the video files are stored on the disk. Previous logs can be acces by clicking Tools->Open Log File Directory in ExPVR.

VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Re: Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 9:41 pm

Ok, I got to looking a bit more at the 320GB drive...

The additional Partition is an ext3fs partition in this case. I accessed it using Ext2Fsd-0.51. That driver is able to mount ext2 and ext3 in read/write but I only mounted it read-only for now. If we get to a point where we want to write to that partition by specifying a custom DB path, this could be one way to do it.

Root Directory contents of the ext3 partition:

Code: Select all

 
 Volume in drive F has no label.
 Volume Serial Number is 5C65-2195

 Directory of F:\

01.01.2000  02:02             2.576 REMBOOK.DBJ
01.01.2000  02:00    <DIR>          THMBCACH
01.01.2000  02:01           718.848 PCAT.DB_EXT_BCK.DB
01.01.2000  02:02            64.248 PCAT.DBJ
01.01.2000  02:01    <DIR>          BCK
01.01.2000  02:01             4.624 PCAT.DB_EXT_BCK.DBJ
01.01.2000  02:02           800.768 PCAT.DB
01.01.2000  02:02            13.928 MFS.DBJ
01.01.2000  02:00    <DIR>          lost+found
01.01.2000  02:00                 0 REMBOOK.DB_EXT_BCK.DB
01.01.2000  02:02             3.072 REMBOOK.DB
01.01.2000  02:00                 0 MFS.DB_EXT_BCK.DB
01.01.2000  02:00                17 PDM.uid
01.01.2000  02:02            47.104 MFS.DB
              11 File(s)      1.655.185 bytes
               3 Dir(s)     435.947.520 bytes free


I used dd for windows from http://www.chrysocome.net/dd to create an image of the partition and I put it in dropbox:
https://www.dropbox.com/s/xoo60zjzljx72 ... ext3fs.zip

Finally, here is the log output from ExPVR:
https://www.dropbox.com/s/klaggdwedectv ... 19-30-.xml

I'll take a look at the 2TB disk now and see if anything is different there.

UPDATE: the 2TB drive seems to have the exact same layout. It's just larger, windows reports the partition as FAT32 and takes quite some time 30-60 seconds to actually mount it. The other partition is ext3 once more and has the same layout as the one on the 320GB disk. Here is the log file from ExPVR when reading in the 2TB drive:
https://www.dropbox.com/s/oxtoxilx0p1qo ... 48-57-.xml

Hope this is useful, let me know if I can help or provide more information...

Cheers,
Lars

VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Re: Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 10:20 pm

One more thing...

mrmt32 wrote:I would expect that the 2TB disk would contain a single 2TB disk, two 1TB 2.5" drives would be far more costly and would require significant software changes. The anytime recordings are stored on the same partition, there is no physical separation of the data. This makes it easy for the box to move recordings from anytime to your planner (it just has to mark the recording as "user" instead of "system" in the planner database).


Does the UK system allow for users to "preserve" Anytime content (i.e. convert it to a regular recording)? This is (sadly) not possible with the german system. However, if this is easily done in the DB, it might be a handy feature :-)

Cheers!
Lars

mrmt32
Site Admin
Posts: 585
Joined: Mon Jul 25, 2011 1:05 am

Re: Trying to use ExPVR with German Sky+

Postby mrmt32 » Fri Apr 12, 2013 10:42 pm

Thanks for sending over those files. From the logs I can see that the format of the XTVFS partition is exactly the same as with the UK boxes (and all the video files are where they should be). I mounted the EXT3 partition in a linux VM and copied out the files, and they are the planner databases. They are of a slightly newer version that the UK ones, but the differences are minor and will not take much to add support for them (the main change is that it seems to use an "Extended Synopsis" field for the synopsis and uses the Synopsis field for a description of the genre).

The hard part now is finding an easy way for ExPVR to deal with the separate partition. Using the Ext2Fsd driver should make things a bit easier (although it is another bit of software that people will have to install).

Adding read support should not be hard at all, I just need to create a popup dialog which lets you point ExPVR to where the databases are saved (which if you have Ext2Fsd should be accessible by windows). The difficulty will come when creating the new disk; I will need to find a way to format an EXT3 partition. The hard way would be to implement this in ExPVR; this would make the process more seamless but would be fairly tricky to code I expect (I have no knowledge of EXT3). I will have to have a look around and see if there is any tool which can do this. I am fairly busy this weekend, but if I get some time I will have a go at getting read support working, that should be a good starting point.

I guess one way round the formatting problem is to require the disk to be prepared in a Sky+ box first, which for these external drives will probably be the case. The Ext2Fsd driver could then just be used in read/write mode.

In the UK it is possible to 'save' on demand showcase programs (what the original 'anytime' on demand service is called, now that an internet-based version exists) by pressing the record button. The program then gets moved over to the personal planner list. Some programs do still expire eventually (they get deleted after a certain time). The expiry time and quota the recording belongs to (the quota can be 'system', 'user', 'recycle' or 'rb') are both in the database however, so it should be possible to 'convert' anytime content to a normal recording. The only possible way this might not work is if there is something built into the encryption of the programs which will stop the card decoding them after a certain time.

VoidPointer
Posts: 8
Joined: Fri Apr 12, 2013 8:07 am

Re: Trying to use ExPVR with German Sky+

Postby VoidPointer » Fri Apr 12, 2013 11:31 pm

Sounds great!

In my case (and I think this is true for most users that are upgrading their german Sky+ boxes) the most common use-case is probably copying contents from the old disk over to the new disk. Ideally, preserving things that have already been recorded on the new disk and "merging" in the content from the old disk. From your explanations that should all work with a mounted ext2 partition and the ability to specify a path to the DB. It would be easier to integrate if there was a library for .NET that would allow you to directly access an ext3fs device but a quick search didn't show anything.


Return to “Help & Support”

Who is online

Users browsing this forum: No registered users and 1 guest