Custom protocol parsing and state-machine reconstruction

Mechanism

Custom services define framing, field widths, encoding, and state transitions beyond the transport layer. Reverse engineering should separate syntax from session semantics. A valid packet in the wrong state may fail even when every field is correctly encoded. Length and checksum fields often cover a particular subset of bytes.

Practical method

Collect several exchanges with one controlled input change per sample. Mark stable headers, lengths, counters, and payload regions. Test byte order and boundary lengths. Implement a parser that rejects incomplete frames, then model the server state transitions independently.

import struct
length = struct.unpack(">H", frame[:2])[0]
assert len(frame[2:]) == length

Interpreting evidence

Validate the model with a new exchange not used to infer it. Predict both bytes and server response state, not only whether the connection remains open.

Pitfalls and limits

A checksum may be calculated before escaping or compression. Message counters can prevent replay. Samples collected from different protocol versions should not be merged into one grammar.