Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory overhead for TransportableObject #1883

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

cjao
Copy link
Contributor

@cjao cjao commented Dec 11, 2023

  • Always represent TransportableObject internally as a single array of bytes. Various properties, such as header, or object_string, decode various segments of the byte array. This allows TransportableObject to be serialized efficiently by simply writing out the byte array.

  • Store the serialized object as raw picklebytes without base64-encoding. As a result, get_deserialized() no longer needs to create a temporary copy of the raw picklebytes. The data segment is directly unpickled. Base64-encoding is applied to the data segment or the entire internal buffer whenever a print friendly representation of the TransportableObject is desired. A backward compatibility layer is introduced to automatically handle TransportableObjects serialized in the earlier format.

  • I have added the tests to cover my changes.
  • I have updated the documentation and CHANGELOG accordingly.
  • I have read the CONTRIBUTING document.

Copy link

codecov bot commented Dec 11, 2023

Codecov Report

Attention: Patch coverage is 97.77778% with 1 line in your changes missing coverage. Please review.

Project coverage is 85.15%. Comparing base (d7841c7) to head (e890bc5).
Report is 19 commits behind head on develop.

Current head e890bc5 differs from pull request most recent head 90b869b

Please upload reports for the commit 90b869b to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1883      +/-   ##
===========================================
+ Coverage    84.60%   85.15%   +0.54%     
===========================================
  Files          295      181     -114     
  Lines        14499    11275    -3224     
  Branches       195        0     -195     
===========================================
- Hits         12267     9601    -2666     
+ Misses        2098     1674     -424     
+ Partials       134        0     -134     
Flag Coverage Δ
Dispatcher 92.55% <ø> (-0.05%) ⬇️
Functional_Tests ?
SDK 79.56% <97.77%> (-0.05%) ⬇️
UI_Backend ?
UI_Frontend ?

@cjao cjao marked this pull request as ready for review December 11, 2023 01:57
@cjao cjao requested a review from a team as a code owner December 11, 2023 01:57
@cjao cjao force-pushed the improve-transportable-overhead branch 3 times, most recently from 9e9caf9 to 3ebee90 Compare January 4, 2024 13:57
@cjao cjao marked this pull request as draft January 4, 2024 14:08
@cjao cjao force-pushed the improve-transportable-overhead branch 2 times, most recently from 3b05bad to e890bc5 Compare February 29, 2024 13:44
@cjao cjao marked this pull request as ready for review April 8, 2024 23:57
@cjao
Copy link
Contributor Author

cjao commented Apr 8, 2024

Tests pass but codecov fails.

@cjao cjao force-pushed the improve-transportable-overhead branch 3 times, most recently from 1f56a35 to e32764f Compare May 16, 2024 00:13
@cjao cjao force-pushed the improve-transportable-overhead branch 3 times, most recently from 4b4a905 to 90b869b Compare June 12, 2024 01:27
* Always represent TransportableObject internally as a single array of
bytes. Various properties, such as `header`, or `object_string`, decode
various segments of the byte array.

* Store the serialized object as raw picklebytes without
base64-encoding. As a result, `get_deserialized()` no longer needs to
create a temporary copy of the raw picklebytes. The data segment is
directly unpickled. Base64-encoding is applied to the data segment or
the entire internal buffer whenever a print friendly representation of
the `TransportableObject` is desired.

* Since the properties of `TransportableObject` are simply views into
the underlying buffer, `TransportableObject` may itself be serialized
efficiently by simply writing out the byte array.
`

Add backward compatibility layer for deserialization
@cjao cjao force-pushed the improve-transportable-overhead branch from 90b869b to ef32837 Compare September 11, 2024 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant