Skip to content

Commit 9da2010

Browse files
authored
Merge pull request #88 from LOLINTERNETZ/fix-sync
Fix sync
2 parents 2689176 + cab2aa7 commit 9da2010

3 files changed

Lines changed: 89 additions & 73 deletions

File tree

README.md

Lines changed: 68 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -2,120 +2,131 @@
22

33
This enables Visual Studio Code's web presence to be mirrored for seamless use in an offline environment (e.g. air-gapped), or to run a private gallery.
44

5-
In effect, content is served through expected interfaces, without changing any of the publicly available binaries. Typically, you would sync the content needing to be availabe on the non-Internet connected system and point the DNS to the mirror service. __No binaries nor extensions are modified.__
6-
5+
In effect, content is served through expected interfaces, without changing any of the publicly available binaries. Typically, you would sync the content needing to be availabe on the non-Internet connected system and point the DNS to the mirror service. **No binaries nor extensions are modified.**
76

87
## Features
98

109
On the Internet connected system , **vscsync** will:
11-
* Mirror the VS Code installer/update binaries across platforms (Windows|Linux|Darwin) and builds (stable|insider);
12-
* Mirror recommended/typical extensions from the marketplace;
13-
* Mirror the malicious extension list;
14-
* Mirror a list of manually specified extensions (artifacts/specified.json); and
15-
* Optionally, mirror all extensions (--syncall, rather than the default of --sync).
10+
11+
- Mirror the VS Code installer/update binaries across platforms (Windows|Linux|Darwin) and builds (stable|insider);
12+
- Mirror recommended/typical extensions from the marketplace;
13+
- Mirror the malicious extension list;
14+
- Mirror a list of manually specified extensions (artifacts/specified.json); and
15+
- Optionally, mirror all extensions (--syncall, rather than the default of --sync).
1616

1717
On the non-Internet connected system, **vscgallery**:
18-
* Implements the updater interface to enable offline updating;
19-
* Implements the extension API to enable offline extension use;
20-
* Implements the malicious extension list;
21-
* Implements initial support for multiple versions;
22-
* Supports extension search (name, author and short description) and sorting;
23-
* Supports custom/private extensions (follow the structure of a mirrored extension); and
24-
* Supports Remote Development.
18+
19+
- Implements the updater interface to enable offline updating;
20+
- Implements the extension API to enable offline extension use;
21+
- Implements the malicious extension list;
22+
- Implements initial support for multiple versions;
23+
- Supports extension search (name, author and short description) and sorting;
24+
- Supports custom/private extensions (follow the structure of a mirrored extension); and
25+
- Supports Remote Development.
2526

2627
Possible TODO List:
27-
* vscgallery - Support paging, if it's really needed (who searches 1000s of extensions anyway).
28-
* Investigate some form of dependency handling (if possible).
29-
* Add test cases.
28+
29+
- vscgallery - Support paging, if it's really needed (who searches 1000s of extensions anyway).
30+
- Investigate some form of dependency handling (if possible).
31+
- Add test cases.
3032

3133
## Requirements
32-
* Docker (ideally with docker-compose for simplicity)
34+
35+
- Docker (ideally with docker-compose for simplicity)
3336

3437
## Getting Started - Full Offline Use - Using Docker Containers
3538

3639
There are two components, **vscsync** which mirrors the content on an Internet connected system, and **vscgallery** which provides the necessary APIs and endpoints necessary to support VS Code's use. While it is designed for offline environments, it is possible, with some DNS trickery, that this could be operated as a "corporate" VS Code gallery.
3740

3841
On the Internet connected system:
3942

40-
1. Acquire/mirror the Docker containers (vscsync/vscgallery).
43+
1. Acquire/mirror the Docker containers (vscsync/vscgallery).
4144

42-
`docker-compose pull`
45+
`docker-compose pull`
4346

4447
2. Setup and run the vscsync service on the Internet connected system.
45-
* Ensuring the artifact directory is accessible to whatever transfer mechanism you will use and vscsync.
46-
* Run vscsync service and ensure the artifacts are generated.
47-
* Wait for the sync to complete. You should see 'Complete' and that it is sleeping when the artifacts have finished downloading.
4848

49-
`docker-compose up vscsync`
49+
- Ensuring the artifact directory is accessible to whatever transfer mechanism you will use and vscsync.
50+
- Run vscsync service and ensure the artifacts are generated.
51+
- Wait for the sync to complete. You should see 'Complete' and that it is sleeping when the artifacts have finished downloading.
5052

51-
4. Copy the artifacts to the non-Internet connected system.
53+
`docker-compose up vscsync`
54+
55+
3. Copy the artifacts to the non-Internet connected system.
5256

5357
On the non-Internet connected system:
5458

5559
1. On the non-Internet connected system, ensure the following DNS addresses are pointed toward the vscgallery service.
56-
* update.code.visualstudio.com
57-
* az764295.vo.msecnd.net
58-
* marketplace.visualstudio.com
5960

60-
This may be achieved using a corporate DNS server, or by modifying a client's host file.
61+
- update.code.visualstudio.com
62+
- az764295.vo.msecnd.net
63+
- marketplace.visualstudio.com
64+
65+
This may be achieved using a corporate DNS server, or by modifying a client's host file.
6166

62-
2. Sort out SSL/TLS within your environment to support offline use.
63-
* Either create a certificate which is signed for the above domains, and is trusted by the clients; or
64-
* Deploy the bundled root and intermediate certificate authority (vscoffline/vscgallery/ssl/ca.crt and ia.crt), with the obvious security tradeoff.
67+
2. Sort out SSL/TLS within your environment to support offline use.
6568

66-
**Windows 10**: Import the certificates into the machine's trusted root certificate authority (Start > "Manage Computer Certificates").
69+
- Either create a certificate which is signed for the above domains, and is trusted by the clients; or
70+
- Deploy the bundled root and intermediate certificate authority (vscoffline/vscgallery/ssl/ca.crt and ia.crt), with the obvious security tradeoff.
6771

68-
**Darwin**: Import the certificates into the machine's trusted root certificate authority.
72+
**Windows 10**: Import the certificates into the machine's trusted root certificate authority (Start > "Manage Computer Certificates").
6973

70-
**Ubuntu**: Easiest method seems to be Open Chrome, navigate to
71-
chrome://settings/certificates, select authorities and add the certificates. Firefox on Ubuntu maintains its own certificate store. Either add the root CA, or switch Firefox to use OS provided certificates (see: https://github.com/LOLINTERNETZ/vscodeoffline/issues/43#issuecomment-1545801875).
74+
**Darwin**: Import the certificates into the machine's trusted root certificate authority.
75+
76+
**Ubuntu**: Easiest method seems to be Open Chrome, navigate to
77+
chrome://settings/certificates, select authorities and add the certificates. Firefox on Ubuntu maintains its own certificate store. Either add the root CA, or switch Firefox to use OS provided certificates (see: <https://github.com/LOLINTERNETZ/vscodeoffline/issues/43#issuecomment-1545801875>).
7278

7379
3. Run the vscgallery service, ensuring the artifacts are accessible to the service. It needs to listen on port 443.
7480

75-
`docker-compose up vscgallery`
81+
`docker-compose up vscgallery`
7682

77-
4. Using Chrome/Firefox navigate to https://update.code.visualstudio.com. You should not see any certificate warnings, if you do it's unlikely to work in VS Code.
83+
4. Using Chrome/Firefox navigate to <https://update.code.visualstudio.com>. You should not see any certificate warnings, if you do it's unlikely to work in VS Code.
7884

7985
5. Open VS Code, hopefully you can magically install extensions and update the install. The Help > Developer Tools > Network should tell you what is going on.
8086

8187
Note: Chrome, rather than other browsers, will likely give you a better indication as to what is going on as VS Code and Chrome share the same certificate trust.
8288

83-
8489
## Getting Started - Standalone Install (Testing or Private Gallery) - Using Docker Containers
85-
This guide will setup the vscsync and vscgallery service on the same Docker host.
90+
91+
This guide will setup the vscsync and vscgallery service on the same Docker host.
8692

8793
1. Grab the docker-compose.yml file.
88-
* Ensure the docker-compose DNS configuration will override what is configured in step 2 (e.g. vscsync can access the Internet, whereas local hosts are directed toward the vscgallery service).
89-
* Ensure both containers will mount the same artifact folder.
94+
95+
- Ensure the docker-compose DNS configuration will override what is configured in step 2 (e.g. vscsync can access the Internet, whereas local hosts are directed toward the vscgallery service).
96+
- Ensure both containers will mount the same artifact folder.
9097

9198
2. Point the DNS addresses to the vscgallery service.
92-
* update.code.visualstudio.com
93-
* az764295.vo.msecnd.net
94-
* marketplace.visualstudio.com
9599

96-
This may be achieved using a corporate DNS server, or by modifying a client's host file.
100+
- update.code.visualstudio.com
101+
- ~~az764295.vo.msecnd.net~~ (Removed 2025/08/22)
102+
- marketplace.visualstudio.com
103+
- main.vscode-cdn.net (Added 2025/08/22)
104+
105+
This may be achieved using a corporate DNS server, or by modifying a client's host file.
97106

98107
3. Deploy SSL/TLS certificates as necessary, as described above.
99108

100109
4. Run the services
101110

102-
`docker-compose up`
111+
`docker-compose up`
103112

104113
5. Using Chrome navigate to https://update.code.visualstudio.com. You should not see any certificate warnings, if you do it's unlikely to work in VS Code.
105114

106115
6. Open VS Code, hopefully you can magically install extensions and update the install. The Help > Developer Tools > Network should tell you what is going on.
107116

108-
109117
## Sync Arguments (vscsync)
110-
These arguments can be passed as command line arguments to sync.py (e.g. --varA or --varB), or passed via the Docker environment variable `SYNCARGS`.
111118

112-
### Typical Sync Args:
113-
* `--sync` To fetch stable binaries and popular extensions.
114-
* `--syncall` To fetch everything (stable binaries, insider binaries and all extensions).
115-
* `--sync --check-insider` To fetch stable binaries, insider binaries and popular extensions.
119+
These arguments can be passed as command line arguments to sync.py (e.g. --varA or --varB), or passed via the Docker environment variable `SYNCARGS`.
116120

117-
### Possible Args:
118-
```
121+
### Typical Sync Args
122+
123+
- `--sync` To fetch stable binaries and popular extensions.
124+
- `--syncall` To fetch everything (stable binaries, insider binaries and all extensions).
125+
- `--sync --check-insider` To fetch stable binaries, insider binaries and popular extensions.
126+
127+
### Possible Args
128+
129+
```text
119130
usage: sync.py [-h] [--sync] [--syncall] [--artifacts ARTIFACTDIR]
120131
[--frequency FREQUENCY] [--check-binaries] [--check-insider]
121132
[--check-recommended-extensions] [--check-specified-extensions]
@@ -138,7 +149,7 @@ optional arguments:
138149
--frequency FREQUENCY
139150
The frequency to try and update (e.g. sleep for '12h'
140151
and try again
141-
--total-recommended N
152+
--total-recommended N
142153
The number of recommended extensions to fetch
143154
(default: 200)
144155
--check-binaries Check for updated binaries
@@ -162,4 +173,4 @@ optional arguments:
162173
--skip-binaries Skip downloading binaries
163174
--debug Show debug output
164175
--logfile LOGFILE Sets a logfile to store loggging output
165-
```
176+
```

vscoffline/sync.py

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,11 @@ def download_update(self, destination):
119119
if os.path.exists(destfile) and vsc.Utility.hash_file_and_check(destfile, self.sha256hash):
120120
log.debug(f'Previously downloaded {self}')
121121
else:
122+
# Some old releases (e.g. stable/win32 - Version: 1.83.1) still reference the old CDN and fail the download,
123+
# so these are skipped.
124+
if self.updateurl.startswith("https://az764295.vo.msecnd.net"):
125+
log.info(f"Skipping old version, no longer available {self}")
126+
return False
122127
log.info(f'Downloading {self} to {destfile}')
123128
result = requests.get(
124129
self.updateurl, allow_redirects=True, timeout=vsc.TIMEOUT)
@@ -372,20 +377,20 @@ def __init__(self, insider, prerelease, version, session):
372377

373378
def get_recommendations(self, destination, totalrecommended):
374379
recommendations = self.search_top_n(totalrecommended)
375-
recommended_old = self.get_recommendations_old(destination)
376-
377-
for extension in recommendations:
378-
# If the extension has already been found then prevent it from being collected again when processing the old recommendation list
379-
if extension.identity in recommended_old.keys():
380-
del recommended_old[extension.identity]
381-
382-
for packagename in recommended_old:
383-
extension = self.search_by_extension_name(packagename)
384-
if extension:
385-
recommendations.append(extension)
386-
else:
387-
log.debug(
388-
f'get_recommendations failed finding a recommended extension by name for {packagename}. This extension has likely been removed.')
380+
# recommended_old = self.get_recommendations_old(destination)
381+
382+
# for extension in recommendations:
383+
# # If the extension has already been found then prevent it from being collected again when processing the old recommendation list
384+
# if extension.identity in recommended_old.keys():
385+
# del recommended_old[extension.identity]
386+
387+
# for packagename in recommended_old:
388+
# extension = self.search_by_extension_name(packagename)
389+
# if extension:
390+
# recommendations.append(extension)
391+
# else:
392+
# log.debug(
393+
# f'get_recommendations failed finding a recommended extension by name for {packagename}. This extension has likely been removed.')
389394

390395
prereleasecount = 0
391396
for recommendation in recommendations:

vscoffline/vsc.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,9 @@
1313
QUALITIES = ["stable", "insider"]
1414

1515
URL_BINUPDATES = r"https://update.code.visualstudio.com/api/update/"
16-
URL_RECOMMENDATIONS = r"https://az764295.vo.msecnd.net/extensions/workspaceRecommendations.json.gz"
16+
URL_RECOMMENDATIONS = r"https://main.vscode-cdn.net/extensions/workspaceRecommendations.json.gz"
1717
URL_MARKETPLACEQUERY = r"https://marketplace.visualstudio.com/_apis/public/gallery/extensionquery"
18-
URL_MALICIOUS = r"https://az764295.vo.msecnd.net/extensions/marketplace.json"
18+
URL_MALICIOUS = r"https://main.vscode-cdn.net/extensions/marketplace.json"
1919

2020
URLROOT = "https://update.code.visualstudio.com"
2121
ARTIFACTS = "/artifacts/"

0 commit comments

Comments
 (0)