Skip to content

Commit 8cae4a8

Browse files
sisongsisong
committed
update version & readme; recode readSyncData() for http download zsync's .gz file;
1 parent 28804c1 commit 8cae4a8

8 files changed

Lines changed: 264 additions & 126 deletions

File tree

‎HDiffPatch‎

Submodule HDiffPatch updated 21 files

‎README.md‎

Lines changed: 114 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
1-
# [hsynz](https://github.com/sisong/hsynz)
2-
[![release](https://img.shields.io/badge/release-v1.2.0-blue.svg)](https://github.com/sisong/hsynz/releases)
1+
# [hsynz]
2+
[![release](https://img.shields.io/badge/release-v1.3.0-blue.svg)](https://github.com/sisong/hsynz/releases)
33
[![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sisong/hsynz/blob/main/LICENSE)
44
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-blue.svg)](https://github.com/sisong/hsynz/pulls)
55
[![+issue Welcome](https://img.shields.io/github/issues-raw/sisong/hsynz?color=green&label=%2Bissue%20welcome)](https://github.com/sisong/hsynz/issues)
66
[![Build Status](https://github.com/sisong/hsynz/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/sisong/hsynz/actions?query=branch%3Amain)
77

88
english | [中文版](README_cn.md)
99

10-
hsynz is a library for delta update using sync algorithm, like [zsync](http://zsync.moria.org.uk).
10+
[hsynz] is a library for delta update using sync algorithm, like [zsync].
1111
rsync over http(s); implement the sync algorithm on client side, and server side only need http(s) cdn. support compressor zstd & libdeflate & zlib, support large file & directory(folder), support multi-thread.
1212

13+
[hsynz] defines its own file format (.hsyni and .hsynz files), and this library is also compatible with the file format of [zsync] (including apply and create .zsync and their .gz files).
14+
15+
[zsync]: http://zsync.moria.org.uk
16+
[hsynz]: https://github.com/sisong/hsynz
1317

1418
Recommended scenarios: Very large number of older versions or where older versions are not available (not saved or modified, etc.) so that all deltas cannot be calculated in advance.
1519

@@ -24,8 +28,11 @@ hsync_demo provides a test client demo for local file testing.
2428
hsync_http provides a download client demo with http(s) support for sync update from a server that provides an http(s) file download service(e.g CDN, support HTTP/1.1 multi range Requests).
2529
Tip: You can also customise other communication methods for sync.
2630

31+
additional, if you have the new file locally & not the old file, but can get a hash certificate file(.hsyni) of the old file,
32+
you can also create a hpatchz format patch file(usage scenario like [rsync]); see the demo cmdline **hsign_diff**.
33+
2734
---
28-
## Compare with [zsync](http://zsync.moria.org.uk)
35+
## Compare with [zsync]
2936
* In addition to supporting source and target as files, support is also provided for directories(folders).
3037
* In addition to supporting compressed release package by zlib; also supported libdeflate & zstd compressor, providing better compression ratio, i.e. smaller downloaded patch package.
3138
* The server-side make support multi-threaded parallel acceleration.
@@ -97,6 +104,15 @@ options:
97104
-C-sha256
98105
-C-crc32
99106
WARNING: crc32 is not strong & secure enough!
107+
-zsync[#KeY#=...#ValuE#=...[#KeY#=...#ValuE#=...]]
108+
create out_hsyni_file(.zsync file format) or out_hsynz_file(.gz file
109+
format) compatible with zsync.
110+
checksum default used sha1 & md4, not need set -C
111+
-s-matchBlockSize size must 2^N; if used -c-gzip or -c-lgzip (out
112+
.gz file), recommend set 4k at most, >=8k may fail.
113+
key-value string pairs will be write in out_hsyni_file; if needed,
114+
you can set Filename,Z-Filename,URL,Z-URL,MTime,Recompress,...
115+
zsync project https://zsync.moria.org.uk
100116
-n-maxOpenFileNumber
101117
limit Number of open files at same time when newDataPath is directory;
102118
maxOpenFileNumber>=8, DEFAULT -n-48, the best limit value by different
@@ -135,6 +151,10 @@ options:
135151
saving diffInfo to cache file for optimize speed when continue sync patch;
136152
-patch#diffFile
137153
local patch(oldPath+diffFile) to outNewPath;
154+
-U
155+
set hsynz_file_url is the original file before compress, please ignore the
156+
compress info in hsyni_file and directly access the data of hsynz file
157+
without decompress.
138158
-cdl-{0|1} or -cdl-{off|on}
139159
continue download data from breakpoint;
140160
DEFAULT -cdl-1 opened, need set -cdl-0 or -cdl-off to close continue mode;
@@ -179,7 +199,7 @@ options:
179199
This cmdline is used for local sync tests, replacing the actual URL remote file with local file, see the hsync_http usage.
180200

181201
---
182-
## hsynz vs [zsync](http://zsync.moria.org.uk):
202+
## [hsynz] vs [zsync]:
183203
case list([download from OneDrive](https://1drv.ms/u/s!Aj8ygMPeifoQgUIZxYac5_uflNoN)):
184204
| |newFile <-- oldFile|newSize|oldSize|
185205
|----:|:----|----:|----:|
@@ -206,43 +226,67 @@ case list([download from OneDrive](https://1drv.ms/u/s!Aj8ygMPeifoQgUIZxYac5_ufl
206226

207227

208228
**test PC**: Windows11, CPU R9-7945HX, SSD PCIe4.0x4 4T, DDR5 5200MHz 32Gx2
209-
**Program version**: hsynz 1.1.1, zsync 0.6.2 (more programs's testing see [HDiffPatch](https://github.com/sisong/HDiffPatch))
229+
**Program version**: hsynz 1.3.0, zsync 0.6.3
210230
**test Program**:
211231
**zsync** run make with `zsyncmake -b 2048 -o {out_newi} {new}`,
212232
client sync diff&patch by `zsync -i {old} -o {out_new} {newi}` (all files are local)
213233
**zsync -z** run make with `zsyncmake -b 2048 -z -u {new.gz} -o {out_newi} {new}`
214234
**hsynz** run make with `hsync_make -s-2k {new} {out_newi} [{-c-?} {out_newz}]`,
235+
run make with `-zsync` means create compatible zsync's file format,
215236
client sync diff&patch by `hsync_demo {old} {newi} {newz} {out_new}` (all files are local)
216237
**hsynz p1** run make without compressor & out_newz , add `-p-1`
217238
**hsynz p8** run make without compressor & out_newz , add `-p-8`
218-
**hsynz p1 -zlib** run make with `-p-1 -c-zlib-9` (run `hsync_demo` with `-p-1`)
219-
**hsynz p8 -zlib** run make with `-p-8 -c-zlib-9` (run `hsync_demo` with `-p-8`)
220-
**hsynz p1 -gzip** run make with `-p-1 -c-gzip-9` (run `hsync_demo` with `-p-1`)
221-
**hsynz p8 -gzip** run make with `-p-8 -c-gzip-9` (run `hsync_demo` with `-p-8`)
222-
**hsynz p1 -ldef** run make with `-p-1 -c-ldef-12` (run `hsync_demo` with `-p-1`)
223-
**hsynz p8 -ldef** run make with `-p-8 -c-ldef-12` (run `hsync_demo` with `-p-8`)
224-
**hsynz p1 -lgzip** run make with `-p-1 -c-lgzip-12` (run `hsync_demo` with `-p-1`)
225-
**hsynz p8 -lgzip** run make with `-p-8 -c-lgzip-12` (run `hsync_demo` with `-p-8`)
226-
**hsynz p1 -zstd** run make with `-p-1 -c-zstd-21-24` (run `hsync_demo` with `-p-1`)
227-
**hsynz p8 -zstd** run make with `-p-8 -c-zstd-21-24` (run `hsync_demo` with `-p-8`)
239+
**hsynz p1 zlib** run make with `-p-1 -c-zlib-9` (run `hsync_demo` with `-p-1`)
240+
**hsynz p8 zlib** run make with `-p-8 -c-zlib-9` (run `hsync_demo` with `-p-8`)
241+
**hsynz p1 gz** run make with `-p-1 -c-gzip-9` (run `hsync_demo` with `-p-1`)
242+
**hsynz p8 gz** run make with `-p-8 -c-gzip-9` (run `hsync_demo` with `-p-8`)
243+
**hsynz p1 ldef** run make with `-p-1 -c-ldef-12` (run `hsync_demo` with `-p-1`)
244+
**hsynz p8 ldef** run make with `-p-8 -c-ldef-12` (run `hsync_demo` with `-p-8`)
245+
**hsynz p1 lgz** run make with `-p-1 -c-lgzip-12` (run `hsync_demo` with `-p-1`)
246+
**hsynz p8 lgz** run make with `-p-8 -c-lgzip-12` (run `hsync_demo` with `-p-8`)
247+
**hsynz p1 zstd** run make with `-p-1 -c-zstd-21-24` (run `hsync_demo` with `-p-1`)
248+
**hsynz p8 zstd** run make with `-p-8 -c-zstd-21-24` (run `hsync_demo` with `-p-8`)
249+
additional tests added **hsign_diff**, which can create hpatchz-compatible patch files using only old data's .hsyni file and the new data.
228250

229251
**test result average**:
230-
|Program|compress|make mem|speed|sync mem|max mem|speed|
231-
|:----|----:|----:|----:|----:|----:|----:|
232-
|zsync|52.94%|1M|353.9MB/s|7M|23M|34MB/s|
233-
|zsync -z|20.67%|1M|14.8MB/s|12M|37M|28MB/s|
234-
|hsynz p1|51.05%|5M|2039.5MB/s|5M|19M|307MB/s|
235-
|hsynz p8|51.05%|21M|4311.9MB/s|12M|27M|533MB/s|
236-
|hsynz p1 zlib|20.05%|6M|17.3MB/s|6M|22M|273MB/s|
237-
|hsynz p8 zlib|20.05%|30M|115.1MB/s|13M|29M|435MB/s|
238-
|hsynz p1 gzip|20.12%|6M|17.3MB/s|6M|22M|268MB/s|
239-
|hsynz p8 gzip|20.12%|30M|115.0MB/s|13M|29M|427MB/s|
240-
|hsynz p1 ldef|19.57%|15M|7.8MB/s|6M|22M|272MB/s|
241-
|hsynz p8 ldef|19.57%|96M|57.0MB/s|13M|29M|431MB/s|
242-
|hsynz p1 lgzip|19.64%|15M|7.9MB/s|6M|22M|267MB/s|
243-
|hsynz p8 lgzip|19.64%|96M|56.9MB/s|13M|29M|419MB/s|
244-
|hsynz p1 zstd|14.96%|532M|1.9MB/s|24M|34M|264MB/s|
245-
|hsynz p8 zstd|14.95%|3349M|10.1MB/s|24M|34M|410MB/s|
252+
|make|mem|speed|patch|compress|mem|max mem|speed|
253+
|:----|----:|----:|:----|----:|----:|----:|----:|
254+
|zsyncmake|1M|381.6MB/s|hsynz p1|45.74%|6M|18M|181MB/s|
255+
|zsyncmake|1M|381.6MB/s|hsynz p8|45.74%|13M|27M|276MB/s|
256+
|zsyncmake|1M|383.4MB/s|`zsync`|45.75%|9M|38M|122MB/s|
257+
|zsyncmake -z|1M|14.8MB/s|hsynz p1|16.56%|6M|21M|169MB/s|
258+
|zsyncmake -z|1M|14.8MB/s|hsynz p8|16.56%|13M|29M|245MB/s|
259+
|zsyncmake -z|1M|14.9MB/s|`zsync`|16.58%|13M|52M|71MB/s|
260+
|hsync_make p1 -zsync|4M|430.9MB/s|hsynz p1|45.39%|6M|19M|187MB/s|
261+
|hsync_make p8 -zsync|14M|591.0MB/s|hsynz p8|45.39%|13M|27M|275MB/s|
262+
|hsync_make p8 -zsync|14M|590.9MB/s|`zsync`|45.39%|9M|37M|119MB/s|
263+
|hsync_make p1 gz -zsync|6M|16.9MB/s|hsynz p1|16.59%|6M|21M|174MB/s|
264+
|hsync_make p8 gz -zsync|29M|101.8MB/s|hsynz p8|16.59%|13M|29M|252MB/s|
265+
|hsync_make p8 gz -zsync|29M|101.8MB/s|`zsync`|16.60%|14M|52M|70MB/s|
266+
|hsync_make p1 lgz -zsync|14M|7.8MB/s|hsynz p1|16.21%|6M|21M|176MB/s|
267+
|hsync_make p8 lgz -zsync|95M|40.3MB/s|hsynz p8|16.21%|14M|29M|252MB/s|
268+
|hsync_make p8 lgz -zsync|95M|40.1MB/s|`zsync`|16.22%|14M|52M|70MB/s|
269+
|hsync_make p1|5M|2179.7MB/s|hsynz p1|44.57%|5M|19M|301MB/s|
270+
|hsync_make p8|12M|3709.2MB/s|hsynz p8|44.57%|13M|27M|478MB/s|
271+
|hsync_make p1 zlib|7M|17.5MB/s|hsynz p1|16.05%|6M|22M|270MB/s|
272+
|hsync_make p8 zlib|30M|105.0MB/s|hsynz p8|16.05%|13M|29M|407MB/s|
273+
|hsync_make p1 ldef|15M|8.0MB/s|hsynz p1|15.68%|6M|22M|270MB/s|
274+
|hsync_make p8 ldef|96M|40.6MB/s|hsynz p8|15.68%|13M|29M|405MB/s|
275+
|hsync_make p1 gz|7M|17.6MB/s|hsynz p1|16.12%|6M|22M|271MB/s|
276+
|hsync_make p8 gz|30M|104.9MB/s|hsynz p8|16.12%|13M|29M|408MB/s|
277+
|hsync_make p1 lgz|15M|7.9MB/s|hsynz p1|15.74%|6M|22M|270MB/s|
278+
|hsync_make p8 lgz|96M|40.6MB/s|hsynz p8|15.74%|13M|29M|405MB/s|
279+
|hsync_make p1 zstd|532M|1.8MB/s|hsynz p1|12.54%|24M|34M|263MB/s|
280+
|hsync_make p8 zstd|3353M|8.1MB/s|hsynz p8|12.54%|23M|34M|384MB/s|
281+
||
282+
|hsign_diff p1|4M|494.7MB/s|hpatchz|43.13%|3M|4M|2409MB/s|
283+
|hsign_diff p8|11M|1412.3MB/s|hpatchz|43.13%|3M|4M|2422MB/s|
284+
|hsign_diff p1 zlib|4M|42.9MB/s|hpatchz|14.78%|4M|4M|926MB/s|
285+
|hsign_diff p8 zlib|13M|246.3MB/s|hpatchz|14.79%|3M|4M|927MB/s|
286+
|hsign_diff p1 ldef|16M|20.1MB/s|hpatchz|14.38%|3M|4M|920MB/s|
287+
|hsign_diff p8 ldef|115M|104.4MB/s|hpatchz|14.38%|3M|4M|917MB/s|
288+
|hsign_diff p1 zstd|205M|8.3MB/s|hpatchz|11.17%|17M|21M|1369MB/s|
289+
|hsign_diff p8 zstd|1348M|17.2MB/s|hpatchz|11.17%|18M|21M|1316MB/s|
246290

247291

248292
## input Apk Files for test:
@@ -288,22 +332,44 @@ case list:
288332
**hsynz ...** make `-s-2k` changed to `-s-1k`
289333

290334
**test result average**:
291-
|Program|compress|make mem|speed|sync mem|max mem|speed|
292-
|:----|----:|----:|----:|----:|----:|----:|
293-
|zsync|62.80%|1M|329.8MB/s|6M|12M|76MB/s|
294-
|zsync -z|59.56%|1M|19.8MB/s|8M|19M|56MB/s|
295-
|hsynz p1|62.43%|4M|1533.5MB/s|4M|10M|236MB/s|
296-
|hsynz p8|62.43%|18M|2336.4MB/s|12M|18M|394MB/s|
297-
|hsynz p1 zlib|58.67%|5M|22.7MB/s|4M|11M|243MB/s|
298-
|hsynz p8 zlib|58.67%|29M|138.6MB/s|12M|19M|410MB/s|
299-
|hsynz p1 gzip|58.95%|5M|22.6MB/s|4M|11M|242MB/s|
300-
|hsynz p8 gzip|58.95%|29M|138.9MB/s|12M|19M|407MB/s|
301-
|hsynz p1 ldef|58.61%|14M|23.7MB/s|4M|11M|242MB/s|
302-
|hsynz p8 ldef|58.61%|96M|149.1MB/s|12M|19M|413MB/s|
303-
|hsynz p1 lgzip|58.90%|14M|23.6MB/s|4M|11M|240MB/s|
304-
|hsynz p8 lgzip|58.90%|96M|149.1MB/s|12M|19M|405MB/s|
305-
|hsynz p1 zstd|57.74%|534M|2.7MB/s|24M|28M|234MB/s|
306-
|hsynz p8 zstd|57.74%|3434M|13.4MB/s|24M|28M|390MB/s|
335+
|make|mem|speed|patch|compress|mem|max mem|speed|
336+
|:----|----:|----:|:----|----:|----:|----:|----:|
337+
|zsyncmake|1M|352.4MB/s|hsynz p1|60.20%|4M|11M|163MB/s|
338+
|zsyncmake|1M|352.4MB/s|hsynz p8|60.20%|12M|19M|228MB/s|
339+
|zsyncmake|1M|353.6MB/s|`zsync`|60.20%|6M|18M|107MB/s|
340+
|zsyncmake -z|1M|20.0MB/s|hsynz p1|57.16%|4M|12M|170MB/s|
341+
|zsyncmake -z|1M|20.0MB/s|hsynz p8|57.16%|12M|20M|240MB/s|
342+
|zsyncmake -z|1M|20.0MB/s|`zsync`|57.16%|9M|25M|79MB/s|
343+
|hsync_make p1 -zsync|4M|412.6MB/s|hsynz p1|60.25%|4M|11M|165MB/s|
344+
|hsync_make p8 -zsync|14M|569.4MB/s|hsynz p8|60.25%|12M|19M|231MB/s|
345+
|hsync_make p8 -zsync|14M|568.6MB/s|`zsync`|60.25%|6M|18M|106MB/s|
346+
|hsync_make p1 gz -zsync|5M|22.2MB/s|hsynz p1|57.24%|4M|12M|169MB/s|
347+
|hsync_make p8 gz -zsync|29M|119.3MB/s|hsynz p8|57.24%|12M|20M|240MB/s|
348+
|hsync_make p8 gz -zsync|29M|119.7MB/s|`zsync`|57.25%|9M|25M|79MB/s|
349+
|hsync_make p1 lgz -zsync|13M|23.2MB/s|hsynz p1|57.18%|4M|12M|168MB/s|
350+
|hsync_make p8 lgz -zsync|96M|111.5MB/s|hsynz p8|57.18%|12M|20M|235MB/s|
351+
|hsync_make p8 lgz -zsync|96M|111.6MB/s|`zsync`|57.19%|9M|25M|77MB/s|
352+
|hsync_make p1|4M|1517.4MB/s|hsynz p1|59.82%|4M|10M|229MB/s|
353+
|hsync_make p8|10M|2200.1MB/s|hsynz p8|59.82%|12M|18M|355MB/s|
354+
|hsync_make p1 zlib|5M|22.9MB/s|hsynz p1|56.17%|4M|11M|242MB/s|
355+
|hsync_make p8 zlib|29M|121.8MB/s|hsynz p8|56.17%|12M|19M|383MB/s|
356+
|hsync_make p1 ldef|14M|24.1MB/s|hsynz p1|56.11%|4M|11M|242MB/s|
357+
|hsync_make p8 ldef|96M|112.0MB/s|hsynz p8|56.11%|12M|19M|382MB/s|
358+
|hsync_make p1 gz|5M|23.1MB/s|hsynz p1|56.44%|4M|11M|242MB/s|
359+
|hsync_make p8 gz|29M|122.3MB/s|hsynz p8|56.44%|12M|19M|382MB/s|
360+
|hsync_make p1 lgz|14M|24.0MB/s|hsynz p1|56.38%|4M|11M|240MB/s|
361+
|hsync_make p8 lgz|96M|111.7MB/s|hsynz p8|56.38%|12M|19M|380MB/s|
362+
|hsync_make p1 zstd|534M|2.6MB/s|hsynz p1|55.22%|24M|28M|234MB/s|
363+
|hsync_make p8 zstd|3443M|10.0MB/s|hsynz p8|55.22%|24M|28M|361MB/s|
364+
||
365+
|hsign_diff p1|3M|392.8MB/s|hpatchz|58.92%|3M|4M|2134MB/s|
366+
|hsign_diff p8|11M|1043.7MB/s|hpatchz|58.92%|3M|4M|2152MB/s|
367+
|hsign_diff p1 zlib|3M|54.0MB/s|hpatchz|54.77%|4M|4M|978MB/s|
368+
|hsign_diff p8 zlib|11M|265.7MB/s|hpatchz|54.77%|4M|4M|972MB/s|
369+
|hsign_diff p1 ldef|15M|53.4MB/s|hpatchz|54.64%|4M|4M|667MB/s|
370+
|hsign_diff p8 ldef|119M|212.4MB/s|hpatchz|54.64%|4M|4M|661MB/s|
371+
|hsign_diff p1 zstd|213M|11.2MB/s|hpatchz|53.89%|20M|20M|1461MB/s|
372+
|hsign_diff p8 zstd|1087M|13.3MB/s|hpatchz|53.89%|20M|20M|1459MB/s|
307373

308374

309375
---

0 commit comments

Comments
 (0)