Updated client with proxy support, synce support, and some prettification. I'm busy for the next few weeks, so I've made it available as 0.2-pre for people to hack on.
IT WORKS! Apparently the 'implied' result code from the device on 0x02 is important, and should basically be AGCLIENT_CONINUE in all cases except the goodbye command, where it should be AGCLIENT_IDLE. Without that, the malsync code got grumpy and reset the connection.
I've successfully updated my device with today's content, using nothing but the cradle and agsync. I've put my source code up for download here for those who are interested. You may have to synchronize a few times, or even force a complete resync of content from the Avantgo configuration panel on your device. Feel free to play, and let me know if you have success!
Updated page. Added more details on commands acquired from reverse-engineering. The code I acquired seems to be so old that the stream protocol has had significant changes. But it does give me several clues as to what to look for.
At this point, I've figured out all the Avantgo commands except for 0x01 and 0x0D, and have tested their operation.
I've got the prototype running successfully, but I'm not having much success because the server isn't sending proper page updates. I'm still trying to figure out why, and will post my results once it works.
AvantGo.com is a company that specializes in synchronization between servers and handheld devices. They offer a free service to the outside world where one can have a handheld device download webpages from an AvantGo server for later offline viewing. The server does translations in some cases to format images for the device, and in some cases, have deals so that companies can provide specialized content for mobile devices.
Most common mobile devices can have Avantgo software installed. With a network connection, the device can sync directly with Avantgo, using the Mobile Application Link protocol. However, the more common mode of synchronization is for a plugin on a desktop computer to perform the synchronization with AvantGo, acting as a middleman between the device and the Internet.
A project called malsync was released under the Mozilla Public License, which provided the ability for non-Windows users to synchronize. However, it was designed only to support Palm Pilot users, and, as far as I can see, does not have code supporting the synchronizaiton process for Pocket PC devices. The AvantGo code is quite complex, and I may not have found it yet if it's there. Please email me if you make a discovery.
I wish to add the ability for Pocket PC users to synchronize with AvantGo through a desktop. My eventual goal is to contribute what I can to the synce project, without which we wouldn't even be experimenting with this.
agsync is a small app that demonstrates a working synchonization process, relying heavily on malsync, and what parts of the protocol I've reverse-engineered.
This is not intended for end-users! It's source-code only, and could well destroy your device for all I know! It synchronized, for me, in my particular environment, at one particular time. That's not a guarantee it will do the same for you. Experiment at your own risk.
Please note, I'm assuming those reading this are fairly familiar with the way Windows CE synchronizes, and the basic protocols used for it.
From a mile up, the synchronization process is as follows:
| 62 00 00 00 | Small-endian 32-bit integer (int32-s). Specifies that 98 bytes are coming after this. |
| 45 00 00 00 | Function code for CeRapiInvoke |
| 00 00 00 00 | Unknown |
| 01 00 00 00 | Unknown |
| 0d 00 00 00 | 13 characters (wide) follow, including null |
| Null terminated wide-character string: malclmgr.dll | |
| 01 00 00 00 | Unknown |
| 14 00 00 00 | 20 characters (wide) follow, including null |
| Null terminated wide-character string: _RAPI_HandleStream2 | |
| 00 00 00 00 | Unknown |
| 01 00 00 00 | Unknown |
The amount of Unknowns in there is saddening. But I'm hoping *hint hint* someone who actually has a Windows box and the equipment to monitor it can make apps with CeRapiInvoke calls and find out what the rest do.
In any case, we do know that somewhere in there must be the fact that we're talking on a stream socket rather than block mode, and also a "parameter block", which is likely some sort of byte array.
On success, this packet will return 00 00 00 00. From this point onwards, the stream will talk in the Avantgo protocol outlined next. If the request fails, an error code is returned, and the stream aborted with the same error code (see below).
As far as I can tell, the bytes A1 A1 A1 A1 followed by a 32-bit value represents closing the stream. I've seen this when Avantgo closes the stream, when you tell Avantgo something confusing, and when you specify an invalid CeRapiInvoke request. The 32-bit value following seems to be a little-endian 32-bit integer result code (0 being success?). Once this happens, the connection is again in RAPI mode.
This is the hard part, as it is the main place where behavior deviates from Palm synchronization. It appears to be a simple command-response system. The desktop sends a single byte as a command code, and then parameters as needed. The device responds with a response code, then data if required. There is very minimal state to worry about in the protocol.
I've outlined some of the command codes I've discovered below. It might help to read details on Avantgo's basic datatypes and Avantgo's complex structures, both extracted from malsync code. Thankfully, this protocol seems to borrow some of the same structures, which makes understanding what's happening much easier.
| Command | Parameters | Responses | Description |
| 0x00 | None | 0x00 - Success | Disconnect. You get a single byte response, and eventually the A1 A1 A1 A1 00 00 00 00 that I suspect is a sign that the stream connection has ended with a result code of 0. |
| 0x01 | CompactInt (0x02 in traces I've seen) - Purpose Unknown. | 0x00 0x02 - Purpose unknown. | I have no idea what this is. Possibly a greeting, to trade IDs or something. Regardless of the specified int, you always get back the same result, as far as I can tell, and it seems not to change any state. It seems to be almost like a separator between sections of things. It happens frequently. Perhaps a ping to verify aliveness after a section of unacknowledged commands? |
| 0x02 | CompactInt cmd byte[] cmdData |
None, as far as I've seen. MAL code implies that it should return a result code somewhere, but traces and experimentation confirm that nothing comes back. | This LOOKS like the command protocol defined in AGProtocol.c/h. The subtypes are specified in great detail there, as well as the format of each individual command. You must have sent 0x07 to open a server before you do this, otherwise, 0x02 will unceremoniously drop your connection. The most common command is 0x10, which is used to add records to the device. |
| 0x03 | AGDBConfig record | 0x00 result code. | Initialize/open a database. The database is automagically closed when a read request finds no more records. |
| 0x04 | None | 0x00 if no record. Otherwise 0x01 If record, AGRecord |
Request record from database. Follows a 0x03. Keep doing it until you get a 0x00 back. Don't do this without a 0x03, because it will cause the stream to be aborted with error code 28 04 00 00. |
| 0x05 | None | 0x00 | Possibly a get-updated-record only thing? As with 0x04, it will abort the stream if you do this without an open database. |
| 0x07 | CompactInt uid | 0x00 (result code) | I suspect 'Start Server', specifying that we are syncing
with a specific server, specified by its uid. This gives me
an error if I specify anything except the UID of a server
known to the device. If the server is not closed, error
code 12 is returned when you end the session.
Opening a server multiple times seems not to be a problem. This conflicts with the old MAL code which indicates that a server and port should be transmitted. Whatever works for you... |
| 0x08 | None | 0x00 (result code) | Close server? This always responds 0x00 even if nothing's open, and traces indicate that it does happen after a server is processed. |
| 0x09 | None | 0x00 - Success CompactInt - size of data to follow AGUserConfig |
Request Avantgo configuration from device. Result is an AGUserConfig, directly from \windows\malconfig.cfg. |
| 0x0A | CompactInt - size of data to follow AGUserConfig |
0x00 - success | Push Avantgo configuration to device. This updates the file
\windows\malconfig.cfg directly. Editor's note: Yes, experimenting with this command can be bad! If you mess up (hint: yeah, I messed up once or twice), you can restore the configuration by copying the file "\Documents and Settings\Username\Application Data\Microsoft\ActiveSync\Profiles\devicename\MAL\MALConfig.cfg" on your desktop to ":\windows\malconfig.cfg" on your device. |
| 0x0B | None | 0x00 - Success AGDeviceInfo |
Request information about the device. This includes things like the OS version, device identifiers, etc. |
| 0x0D | None | 0x00 3xCompactInt. Uses unknown. |
Unknown. This is used very early. Nothing seems to really affect it much |
| Invalid ops | None | These are SILENTLY IGNORED! |
That's not all the commands. These are just the ones I was able to quickly figure out and explain. More will come in time.
| boolean | 8-bit signed value. 0 == FALSE, > 0 == TRUE, < 0 is invalid. |
| byte | 8-bit unsigned value. Interpretation varies. |
| int8 | Signed 8-bit integer. |
| int16 | Big-endian 16-bit value. |
| int32 | Big-endian 32-bit value. |
| CompactInt | Read byte. If < 254, value is the byte you just read. If == 254, read an int16, return that. IF == 255, read an int32, return that. |
| Array (note my notation: type[]) | Read CompactInt. This is the array length. Read this many following "type" elements. |
| CString | Null-terminated ASCII string. |
| String | byte[] representing an ASCII string |
Please note, most of the complex datatypes can be derived from the "Read" functions for them in Malsync, so my summaries will be brief. Also, since most of this is ripped indirectly from malsync, these structures are also released under the Mozilla Public License.
struct AGUserConfigData {
int16 Signature= 0xDEAA;
int16 Version= 0x0000;
CompactInt uid;
CompactInt flags; // (suspected not used, normally 0x00)
CompactInt[] uidDeletes;
AGServerConfig[] servers;
CompactInt expansion1= 0, expansion2= 0, expansion3= 0, expansion4= 0;
byte[] reserved= {};
}
struct AGServerConfig {
int16 Signature= 0xDEAA;
int16 Version= 0x0000;
CompactInt uid;
CompactInt status;
CString server;
CompactInt port;
CString username;
CString clearTextPassword; // Thankfully, blank
// Note: this uses an int8 instead of a compactInt for length
byte[] nonce;
boolean disabled;
CString friendlyName;
CString serverType; // "AvantGo"
CString userURL;
CString description;
CString serverUri;
byte[] sequenceCookie; // 5 bytes on my implementation
AGDBConfig[] databases;
boolean sendDeviceInfo;
int8 hashPassword;
CompactInt connectTimeout;
CompactInt writeTimeout;
CompactInt readTimeout
boolean connectSecurely;
boolean allowSecureConnection;
CompactInt flags; // TODO: Discover me!
CompactInt expansion1= 0, expansion2= 0, expansion3= 0, expansion4= 0;
byte[] reserved= {};
}
struct AGDBConfig {
int16 Signature= 0xD5AA;
int16 Version= 0x0000;
CString dbname; /* These can be matched to the subdirectories of the
directory specified in
HKEY_CURRENT_USER\Software\AvantGo\DatabaseLocation */
CompactInt type;
boolean sendRecordPlatformData;
byte[] platformData;
int32[] newIds;
CompactInt expansion1= 0, expansion2= 0, expansion3= 0, expansion4= 0;
byte[] reserved= {};
}
struct AGRecord {
CompactInt uid;
enum AGRecordStatus status= {UNMODIFIED= 0, UPDATED= 1, NEW= 2,
DELETED= 3, NEW_TEMP= 4};
byte[] recordData;
byte[] platformData;
}
struct AGDeviceInfo {
int32 availableBytes; // Yup, free storage.
int32 screenWidth, screenHeight, colorDepth;
int32 platformDataLen
Bytes platformData;
CString osName;
CString OSVersion;
CString setLanguage, charset, serialnumber;
}