fix: detect completed downloads by checking final file exists (#1582)
## Summary Split from #1547 per review feedback. When scanning existing download status, `get_downloaded_size()` returns bytes from either the final file or its `.partial` counterpart — so a `.partial` file with all bytes downloaded (e.g. process killed before hash verification and rename) could be falsely reported as complete. The previous approach (commit 3b54e7df) added a byte-comparison fallback in the coordinator (`downloaded >= total > 0`), but this suffered from the same `.partial` conflation issue. **Fix:** Check whether the final (non-`.partial`) file actually exists on disk before marking status as `"complete"` in `download_utils.py`. This is the only reliable signal that hash verification passed and the rename from `.partial` succeeded. The coordinator-level byte comparison is removed since the source now reports correctly. ### Changes - `download_utils.py`: Add `final_file_exists` check — only report `"complete"` when the renamed, hash-verified file exists with matching size - `coordinator.py`: Revert to simple `progress.status == "complete"` check, removing the byte-comparison fallback **Note:** The corresponding byte-comparison workaround in #1547 should also be removed. ## Test plan - [x] basedpyright — 0 errors - [x] ruff check — passes - [x] pytest — 218 passed, 1 skipped (1 pre-existing Rust bindings failure) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -823,6 +823,7 @@ async def download_shard(
|
||||
|
||||
for file in filtered_file_list:
|
||||
downloaded_bytes = await get_downloaded_size(target_dir / file.path)
|
||||
final_file_exists = await aios.path.exists(target_dir / file.path)
|
||||
file_progress[file.path] = RepoFileDownloadProgress(
|
||||
repo_id=shard.model_card.model_id,
|
||||
repo_revision=revision,
|
||||
@@ -832,7 +833,9 @@ async def download_shard(
|
||||
total=Memory.from_bytes(file.size or 0),
|
||||
speed=0,
|
||||
eta=timedelta(0),
|
||||
status="complete" if downloaded_bytes == file.size else "not_started",
|
||||
status="complete"
|
||||
if final_file_exists and downloaded_bytes == file.size
|
||||
else "not_started",
|
||||
start_time=time.time(),
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user