Brainstorm: `url.parse()` -> `new url`

[AugustinMauroy]

Description

This codemod should migrate url.parse() to new url.

• It's should handle both require() function and import statement.
• It's should support both url node:url specifier.
• If the url module isn't use elsewhere we should remove the importation.
• auth property should be replaced by ${modernURL.username}:${modernURL.password}
• path property should be replaced by ${modernURL.pathname}${modernURL.search}
• hostname property should be replaced by modernURL.hostname.replace(/^\[|\]$/, '') to remove the brackets for IPv6 addresses.

Example

In these example we use legacyURL to refer to the object returned by url.parse(), and modernURL to refer to the object created with new URL().
In the codemod implementation, it' will be the name of the variable that is used by the user so it's shouldn't be changed.

Case 1: Basic usage

Before:

const url = require('node:url');

const legacyURL = url.parse('https://example.com/path?query=string#hash');

After:

const modernURL = new URL('https://example.com/path?query=string#hash');

Case 2: auth property

Before:

const url = require('node:url');

const legacyURL = url.parse('https://user:pass@example.com/path');

const { auth } = legacyURL;
const urlAuth = legacyURL.auth;

After:

const modernURL = new URL('https://user:pass@example.com/path');

const auth  = `${modernURL.username}:${modernURL.password}`;
const urlAuth = `${modernURL.username}:${modernURL.password}`;

Case 3: path property

Before:

const url = require('node:url');

const legacyURL = url.parse('https://example.com/path?query=string#hash');

const { path } = legacyURL;
const urlPath = legacyURL.path;

After:

const modernURL = new URL('https://example.com/path?query=string#hash');

const path = `$(modernURL.pathname}${modernURL.search}`;
const urlPath = `${modernURL.pathname}${modernURL.search}`;

Case 4: hostname property with IPv6 address

Before:

const url = require('node:url');

const legacyURL = url.parse('https://[2001:db8::1]/path');

const { hostname } = legacyURL;
const urlHostname = legacyURL.hostname;

After:

const modernURL = new URL('https://[2001:db8::1]/path');

const hostname = modernURL.hostname.replace(/^\[|\]$/, '');
const urlHostname = modernURL.hostname.replace(/^\[|\]$/, '');

Case 5: cjs url module elsewhere

Before:

const { parse, foo } = require('node:url');

const legacyURL = parse('https://example.com/path?query=string#hash');

foo(); // does something else

After:

const { foo } = require('node:url');

const modernURL = new URL('https://example.com/path?query=string#hash');

foo(); // does something else

Case 6: import statement

Before:

import { parse } from 'node:url';

const legacyURL = parse('https://example.com/path?query=string#hash');

After:

import { URL } from 'node:url';

const modernURL = new URL('https://example.com/path?query=string#hash');

Case 7: import statement with other named exports

Before:

import { parse, foo } from 'node:url';

const legacyURL = parse('https://example.com/path?query=string#hash');

foo(); // does something else

After:

import { URL, foo } from 'node:url';

const modernURL = new URL('https://example.com/path?query=string#hash');

foo(); // does something else

REFS

https://nodejs.org/dist/docs/api/url.html
https://url.spec.whatwg.org

[AugustinMauroy]

I had make some research on the topic, here a test suite that compare both and highlights the points that could be blocked

Test Suite

    const url = require('node:url');
    const assert = require('node:assert');
    const { describe, it } = require('node:test');

    describe('legacy URL and modern URL compatibility', () => {
    // Test basic URL
    it('should handle basic URL', () => {
        const legacyURL = url.parse('https://example.com/path?query=string#hash');
        const modernURL = new URL('https://example.com/path?query=string#hash');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname + modernURL.search); // warning: legacyURL.path does not include search
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
        assert.strictEqual(legacyURL.search, modernURL.search);
        assert.strictEqual(legacyURL.hash, modernURL.hash);
        assert.ok(modernURL.origin);
        assert.ok(modernURL.href);
    });

    // Test URL with authentication
    it('should handle URL with authentication', () => {
        const legacyURL = url.parse('https://user:pass@example.com/path');
        const modernURL = new URL('https://user:pass@example.com/path');

        assert.strictEqual(legacyURL.auth, modernURL.username + ':' + modernURL.password); // warning: legacyURL.auth is not available in modern URL
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
    });

    // Test URL with port
    it('should handle URL with port', () => {
        const legacyURL = url.parse('https://example.com:8080/path');
        const modernURL = new URL('https://example.com:8080/path');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.port, modernURL.port);
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
    });

    // Test URL with no path
    it('should handle URL with no path', () => {
        const legacyURL = url.parse('https://example.com');
        const modernURL = new URL('https://example.com');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
    });

    // Test URL with query parameters
    it('should handle URL with query parameters', () => {
        const legacyURL = url.parse('https://example.com/path?param1=value1&param2=value2');
        const modernURL = new URL('https://example.com/path?param1=value1&param2=value2');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname + modernURL.search); // warning: legacyURL.path does not include search
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
        assert.strictEqual(legacyURL.search, modernURL.search);
    });

    // Test URL with hash
    it('should handle URL with hash', () => {
        const legacyURL = url.parse('https://example.com/path#section1');
        const modernURL = new URL('https://example.com/path#section1');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
        assert.strictEqual(legacyURL.hash, modernURL.hash);
    });

    // Test URL with special characters
    it('should handle URL with special characters', () => {
        const legacyURL = url.parse('https://example.com/path?query=string with spaces#hash with spaces');
        const modernURL = new URL('https://example.com/path?query=string with spaces#hash with spaces');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname + modernURL.search); // warning: legacyURL.path does not include search
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
        assert.strictEqual(legacyURL.search, modernURL.search);
        assert.strictEqual(legacyURL.hash, modernURL.hash);
    });

    // Test URL with different protocols
    it('should handle URL with different protocols', () => {
        const protocols = ['http:', 'https:', 'ftp:', 'file:'];
        protocols.forEach(protocol => {
        const legacyURL = url.parse(`${protocol}//example.com/path`);
        const modernURL = new URL(`${protocol}//example.com/path`);

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname);
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
        });
    });

    // Test URL with IPv6 address
    it('should handle URL with IPv6 address', () => {
        const legacyURL = url.parse('https://[2001:db8::1]/path');
        const modernURL = new URL('https://[2001:db8::1]/path');

        assert.strictEqual(legacyURL.protocol, modernURL.protocol);
        assert.strictEqual(legacyURL.host, modernURL.host);
        assert.strictEqual(legacyURL.hostname, modernURL.hostname.replace(/^\[|\]$/, '')); // warning: legacyURL.hostname does not include brackets
        assert.strictEqual(legacyURL.path, modernURL.pathname);
        assert.strictEqual(legacyURL.pathname, modernURL.pathname);
    });
    });

[mcollina]

A codemod could handle maybe 80% of the cases - which I think would be plenty.

Maybe we could have a url.parseWithNative that emits the same data structure but with the new behavior.

[AugustinMauroy]

A codemod could handle maybe 80% of the cases - which I think would be plenty.

I would like to know the 20% so I can figure out how to shape it.

Maybe we could have a url.parseWithNative that emits the same data structure but with the new behavior.

We agree that this idea has nothing to do with codemod; it's just for node-core, right?

[alexbit-codemod]

@AugustinMauroy, i used Codemod AI to generate this: https://app.codemod.com/studio?share_id=512f5c98241325a2756d71
maybe you can help me take it to the next level for tricker cases.

btw, there are two tabs in the studio link above:
Query: ast-grep YAML rule, great for pattern detection and simple transformations.
Codemod: ast-grep JS/napi, great for more sophisticated transformations.

ping me if you have follow up questions on this.

[styfle]

A codemod could handle maybe 80% of the cases - which I think would be plenty.

I would like to know the 20% so I can figure out how to shape it.

The biggest difference is that url.parse() supports relative urls and new URL() does not.

url.parse('/foo?v=1') // works fine

new URL('/foo?v=1') // throws an error

That alone makes the codemod challenging.