r1400 - WM 6.5 - ItemOperations Fetch fails

[liverpoolfcfan]

Andreas/Manfred,

I have found a difference in the output WBXML between zpush2 and as12.1 that may be causing the problem.

If I am reading it correctly the Part number in the response should be ZERO offset according to the MS spec.

The WBXML response, which contains status and application data for all requested items. The WBXML response is always the first part in the response. Items composed of binary content have a Part element (section 2.2.3.120) that indicates the index (base 0) of the body part that corresponds to that item in the multipart structure. This index is used by the client to find the appropriate [start, count] entry in the metadata.

but zpush2 is outputting Part 1 (ONE) for the file.

Can you tell me what change I would need to make to have the attachment referred to as part 0 (ZERO) - and I will test it.

Here is the debug log of WM6.5 requesting a PDF attachment from a z-push-2 server - Note the part 1 in the response. The phone never finishes "downloading"

It is followed by the same request to an as12.1 server. Note the part 0 in the response. The phone gets the file and can open it correctly.


z-push-2 logs for ItemOperations Fetch

Code: Select all

23/07/2012 15:23:12 [27311] [vincents] [WBXML] O  <ItemOperations:ItemOperations>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O   <ItemOperations:Status>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O   1
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O   </ItemOperations:Status>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O   <ItemOperations:Response>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O    <ItemOperations:Fetch>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     <ItemOperations:Status>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     1
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     </ItemOperations:Status>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     <AirSyncBase:FileReference>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     2:95844:2
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     </AirSyncBase:FileReference>
23/07/2012 15:23:12 [27311] [vincents] [DEBUG] START GetAttachmentData { attname = '2:95844:2' }
23/07/2012 15:23:12 [27311] [vincents] [DEBUG] END GetAttachmentData { application/pdf }
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     <ItemOperations:Properties>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      <AirSyncBase:ContentType>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      application/pdf
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      </AirSyncBase:ContentType>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      <ItemOperations:Part>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      1
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O      </ItemOperations:Part>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O     </ItemOperations:Properties>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O    </ItemOperations:Fetch>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O   </ItemOperations:Response>
23/07/2012 15:23:12 [27311] [vincents] [WBXML] O  </ItemOperations:ItemOperations>


Here is the equivalent attachment requested through as12.1 - note that it is referred to as Part 0 (ZERO)

as12.1 logs for ItemOperations Fetch

Code: Select all

07/23/12 13:24:36 [24122] O  <ItemOperations:ItemOperations>
07/23/12 13:24:36 [24122] O   <ItemOperations:Status>
07/23/12 13:24:36 [24122] O   1
07/23/12 13:24:36 [24122] O   </ItemOperations:Status>
07/23/12 13:24:36 [24122] O   <ItemOperations:Response>
07/23/12 13:24:36 [24122] O    <ItemOperations:Fetch>
07/23/12 13:24:36 [24122] O     <ItemOperations:Status>
07/23/12 13:24:36 [24122] O     1
07/23/12 13:24:36 [24122] O     </ItemOperations:Status>
07/23/12 13:24:36 [24122] O     <AirSyncBase:FileReference>
07/23/12 13:24:36 [24122] O     2:95844:2
07/23/12 13:24:36 [24122] O     </AirSyncBase:FileReference>
07/23/12 13:24:36 [24122] START ItemOperationsGetAttachmentData { attname = '2:95844:2' }
07/23/12 13:24:36 [24122] END ItemOperationsGetAttachmentData { application/pdf }
07/23/12 13:24:36 [24122] O     <ItemOperations:Properties>
07/23/12 13:24:36 [24122] O      <AirSyncBase:ContentType>
07/23/12 13:24:36 [24122] O      application/pdf
07/23/12 13:24:36 [24122] O      </AirSyncBase:ContentType>
07/23/12 13:24:36 [24122] O      <ItemOperations:Part>
07/23/12 13:24:36 [24122] O      0
07/23/12 13:24:36 [24122] O      </ItemOperations:Part>
07/23/12 13:24:36 [24122] O     </ItemOperations:Properties>
07/23/12 13:24:36 [24122] O    </ItemOperations:Fetch>
07/23/12 13:24:36 [24122] O   </ItemOperations:Response>
07/23/12 13:24:36 [24122] O  </ItemOperations:ItemOperations>


[mku]

Hi Vincent,

this is a little bit strange because with <ItemOperations:Part>1</ItemOperations:Part> it works fine with my Windows Phone 7.5. And also the MS documentation says <Part>1</Part> a little above the part you've quoted.

If you want to test it, search for

Code: Select all

$encoder->content(count($encoder->getBodypartsCount()));

in streamer.php (should be line 287) and change it to

Code: Select all

$encoder->content(count($encoder->getBodypartsCount()) - 1);

Greets, Manfred

[liverpoolfcfan]

In wbxmlencoder.php function processMultipart()

Code: Select all

        foreach ($this->bodyparts as $bp) {
            $blockstart = $blockstart + $len;
            $len = fstat($bp);
            $len = (isset($len['size'])) ? $len['size'] : 0;
            $data .= pack("ii", $blockstart, $len);
        }


when I debug this part of the code - $len = fstat($bp) returns NOTHING ... Do you expect that ? Do you get the same ?

For reference in the backend I am creating the stream as follows. Could there be a problem with the way I am opening the stream ? Note that it does of course work fine for GetAttachment() calls from Nokia and other phones.

Code: Select all

             $attachment = new SyncItemOperationsAttachment();

            $url = ZIMBRA_URL . "/service/content/get?id=" . $id;
            if ($part != "") {
                $url = $url . "&part=" . $part;
            }
         
//debugLog( 'Url ['.$url.']' );

            $options = array( 'http' => array( 'method' => 'GET',
                                               'header' => 'Cookie: ZM_AUTH_TOKEN=' . $this->_authtoken )
                       );
            $context = stream_context_create($options);
            $stream = fopen( $url, 'r', false, $context );


            $attachment->data = $stream;
          $attachment->contenttype = trim($msg['ct']);
            debugLog("END GetAttachmentData { ". $attachment->contenttype ." }");
            return $attachment;


[mku]

Hi Vincent,

in your stream wrapper class you have to add stream_stat function which returns the array with stat information (check the PHP documentation for stat if you need some more information). E.g this is how it looks like in stringstreamwrapper.php ($this->stringlength is set in stream_open):

Code: Select all

    /**
    * Retrieves information about a stream
    *
    * @access public
    * @return array
    */
    public function stream_stat() {
        return array(
            7               => $this->stringlength,
            'size'          => $this->stringlength,
        );
    }

Greets, Manfred

[liverpoolfcfan]

I have confirmed the lack of the $len is the problem.

I added some code to parse the stream_get_meta_data for a PDF request - where it contained a Content-Length header. Once it got the length - it was able to open the attachment.

I have never written a stream wrapper - so am a bit stumped with this now. The access to the MIME PARTS on Zimbra is through http(s) GET so I just used the standard out of the box HTTP context to wrap the call

Code: Select all

            $context = stream_context_create($options);
            $stream = fopen( $url, 'r', false, $context );


How would I go about replacing this with a stream wrapper where I could add a stream_stat function ?


Or, alternatively, is there a way the size could be supplied instead as a non-streamed property of the SyncItemOperationsAttachment class ? This would allow for more flexibility for custom backends.

When setting up the stream in the backend I have access to the MIME data and can get the size of the attachment. However, as it is not a straightforward file - it does not seem to carry a size with it when streamed.

Thanks,
Vincent

[mku]

Hi Vincent,

having everything in a custom stream wrapper is nicer and you're also more flexible as adding a non-streaming property.

I'll modified StringStreamWrapper a bit as an example for a custom stream wrapper.

Code: Select all

class CustomStreamWrapper {
    const PROTOCOL = "customstream";

    private $customstream;
    private $position;
    private $streamlength;
    private $streamdata;

    /**
     * Opens the stream
     * The string to be streamed is passed over the context
     *
     * @param string    $path           Specifies the URL that was passed to the original function
     * @param string    $mode           The mode used to open the file, as detailed for fopen()
     * @param int       $options        Holds additional flags set by the streams API
     * @param string    $opened_path    If the path is opened successfully, and STREAM_USE_PATH is set in options,
     *                                  opened_path should be set to the full path of the file/resource that was actually opened.
     *
     * @access public
     * @return boolean
     */
    public function stream_open($path, $mode, $options, &$opened_path) {
        $contextOptions = stream_context_get_options($this->context);
        if (!isset($contextOptions[self::PROTOCOL]['customstream']))
            return false;

        $this->position = 0;

        // this is our stream!
        $this->customstream = $contextOptions[self::PROTOCOL]['customstream'];

        $this->streamdata = stream_get_meta_data($this->customstream);
        $this->streamlength = $this->streamdata['wrapper_data'][9]; //you might have to change this
        ZLog::Write(LOGLEVEL_DEBUG, sprintf("CustomStreamWrapper::stream_open(): initialized stream length: %d", $this->streamlength));

        return true;
    }

    /**
     * Reads from stream
     *
     * @param int $len      amount of bytes to be read
     *
     * @access public
     * @return string
     */
    public function stream_read($len) {
        //you might have to modify this to get the attachment's data
        $data = substr($this->stringstream, $this->position, $len);
        $this->position += strlen($data);
        return $data;
    }

    /**
     * Returns the current position on stream
     *
     * @access public
     * @return int
     */
    public function stream_tell() {
        return $this->position;
    }

   /**
     * Indicates if 'end of file' is reached
     *
     * @access public
     * @return boolean
     */
    public function stream_eof() {
        return ($this->position >= $this->streamlength);
    }

    /**
    * Retrieves information about a stream
    *
    * @access public
    * @return array
    */
    public function stream_stat() {
        return array(
            7               => $this->streamlength,
            'size'          => $this->streamlength,
        );
// here you can also add other properties of stat() if you want to
    }

   /**
     * Instantiates a CustomStreamWrapper
     *
     * @param string    $string     The string to be wrapped
     *
     * @access public
     * @return CustomStreamWrapper
     */
     static public function Open($stream) {
        $context = stream_context_create(array(self::PROTOCOL => array('customstream' => &$stream)));
        return fopen(self::PROTOCOL . "://",'r', false, $context);
    }
}

stream_wrapper_register(CustomStreamWrapper::PROTOCOL, "CustomStreamWrapper")

Then in your code you have something like:

Code: Select all

$attachment->data = CustomStreamWrapper::Open($attachmentdata);

That should do it and you don't have to care about the streamer and WBXML.

If it helps, I'll put this as a separate posting in the developer section.

Greets, Manfred

[liverpoolfcfan]

Thanks a million for this. I will try it out later or probably tomorrow at this point.

I did see that the length was not always available in the HTTP context - so not sure if the custom wrapper will act differently.

If it doesn't, could I add an extra parameter to the ::Open function, and to the stream_open function to pass in the length - OR - do those functions have to have the specified number of parameters ?

Thanks,
Vincent

[liverpoolfcfan]

Hi Manfred,

Firstly, thanks again for helping.

I tried this out but it didn't work. I got a "not a valid stream" error - followed by endless eof() messages - it looped in wbxmlencoder until I killed HTTPD basically. The problem may be that it does not allow me to enter the required authorization. Perhaps there is some error condition checking needs to be added in there to prevent loops.

Now, back to the challenge in hand.

I did succeed in getting the attachment to stream to the mobile if I broke the process into two parts.

1. Use an HTTPS:// GET to retrieve the attachment data into an object in the backend.
2. Use StringStreamWrapper to stream the attachment data from the object out to z-push

However, doesn't this break the entire intention of a stream ? I thought the idea of a stream was that the code in the middle did not have to handle the large object - rather it opened a pointer to it and handed it off to allow the backend data to be streamed through to the client without having to be instantiated in the middle. Am I misinterpreting the idea of a stream ?

Assuming the goal is to stream the data directly from the backend to the client - then this is what I need to be able to do.

1. Create a custom context HTTPS stream to pass an authorization cookie to the backend to allow me to get the data, and
2. Somehow wrap that stream in a way that I can set the stream length as it is not always available in the meta_data from the backend (for example I have found that .TXT files do not come across with a content-length set). I know the length as it is available to me in the mime data - I just need a way to get it into the wrapper class.

I hope this makes sense.
Vincent

[mku]

Hi Vincent,

as the stream wrappers are quite stand-alone, you could add parameters to ::Open.

Yes, the idea of the stream wrapper is to have a pointer to it and stream it to the device without keeping it in memory.

I don't know anything about Zimbra backend and how's getting attachments is working there, so I can't help you on that.

Greets, Manfred