Support multiple buckets

[atrauzzi]

If I'm correct in reading the current code, it looks like the first bucket configured in php.ini is the one that this package uses.

Would it be worthwhile to allow the bucket used to be configured at a driver/L5 filesystem level?

[shpasser]

As of now the first enabled GSC bucket is used for storage_path(). By default GAE Filesystem wrapper uses storage_path().'/app' as its root. storage_path() is determined according to the environment the app is running on, i.e. GAE or non-GAE, and will work properly in local and production environments. If a specific filesystem wrapper is configured with some GCS bucket path it will fail in local environment, although it will work in production.

[shpasser]

But I could convert disk root path from gs://{$bucket}/some_path to storage_path()."/{$bucket}/some_path" when running not on GAE.

[shpasser]

Any thoughts?

[atrauzzi]

I'm kind of torn between the simplicity of it all just being file paths, but the explicitness of using straight-up bucket names.

Something I did in my build setup (which is not optimal) is run my php.ini through sed. This has given me the ability to use a dns-named bucket for static hosting (as opposed to the default one GAE creates).

That prompted this: https://code.google.com/p/googleappengine/issues/detail?id=12297

Based on all this though, I'm inclined to think that we should have somewhere in our applications an environment variable that defines which bucket we want to use. Which then gets picked up by configurations of the gae filesystem driver. Whether it's #default# or a specific name.

Hopefully what I said makes sense or has some valuable takeaways ;)

[shpasser]

Does this help?

    /**
     * Override the storage path
     *
     * @return string Storage path URL
     */
    public function storagePath()
    {
        if ($this->runningOnGae)
        {
            if ( ! is_null($this->gaeBucketPath))
            {
                return $this->gaeBucketPath;
            }

            $buckets = ini_get('google_app_engine.allow_include_gs_buckets');

            // Get the defined bucket or the first bucket in the list.
            $bucket = env('GAE_GCS_BUCKET', current(explode(', ', $buckets)));

            if ($bucket)
            {
                $this->gaeBucketPath = "gs://{$bucket}/storage";

                if (env('GAE_SKIP_GCS_INIT'))
                {
                    return $this->gaeBucketPath;
                }

                if ( ! file_exists($this->gaeBucketPath))
                {
                    mkdir($this->gaeBucketPath);
                    mkdir($this->gaeBucketPath.'/app');
                    mkdir($this->gaeBucketPath.'/framework');
                    mkdir($this->gaeBucketPath.'/framework/views');
                }

                return $this->gaeBucketPath;
            }
        }

        return parent::storagePath();
    }

[atrauzzi]

$bucket = env('GAE_GCS_BUCKET', current(explode(', ', $buckets)));

Looks perfect.

[shpasser]

I will have to rearrange some of the documentation and add the environment variable to app.yaml (because it is needed before .env is loaded). Will keep you posted.

BTW, what did you mean by Is cachefs being phased out? (on gitter)

[atrauzzi]

Oh, sorry. I meant to follow up there, it's probably nothing. I just had seen some directories and files (services.json) being created in cloud storage that I thought were things going to cachefs. And it looks like they still are.

[shpasser]

Please notice that GAE_GCS_BUCKET variable will be added to app.yaml. Let me know if that's Ok.

Regarding services.json, I couldn't see any problems with my test application. The file is created in GCS when CACHE_SERVICES_FILE is false, but when CACHE_SERVICES_FILE is true it is created in cachefs(memcached). Please open an issue if there is a problem.

[atrauzzi]

Does GAE_GCS_BUCKET need to be in app.yaml to be available in time? It might be nicer to have it in .env if possible.

[shpasser]

When storagePath() first called .env is not yet loaded, besides the setting itself is GAE-only, why not put it in GAE related configuration file?

Anyway, I'm open to suggestions.

[shpasser]

Actually, I can return an instance of LazyString containing a callback to actual storage path initialization instead of the storage path string. Then I will be able to use environment variables stored in .env

class LazyString
{
    protected $getString;

    // Store a callback to storage path initialization function
    public function __construct(callable $getString)
    {
        $this->getString = $getString;
    }

    // Initialize the storage path and return its string value
    public function __toString()
    {
        return call_user_func($this->getString);
    }
}

It works, but is this bit of 'ease of use' worth the complexity of the solution?

[atrauzzi]

I guess the issue is that the bucket name is environment specific for me. I have staging and production buckets to ensure isolation. The only file I'd ideally like to touch during my build process is .env. Putting it in the .yaml would mean that all deployments have to use the same bucket. Unless I sed it. But that's honestly something I'm trying to get away from.

I have my request in with GAE to be able to select the default bucket for an app, which ought to cover 99% of the scenarios by setting it to #default#. But we're not there yet, and it is conceivable that someone could still have multiple buckets they want to use, even outside of any default.

[shpasser]

Just trying to understand, you would like to touch only .env file during the build process.
On the other hand if #default# stuff will be implemented by GAE you'll probably have an option to set the default GSC bucket via php.ini. Then how does the last one solve the problem?

[atrauzzi]

php.ini is unavoidable because it's dictated by app engine. What would be ideal though is ignore the default and have laravel use exactly the bucket I want which will be set in .env.

#default# doesn't matter anymore at that point. It can be set to whatever it wants, my app at runtime will just choose the correct bucket to operate on based on my settings.