SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Habemus codex et ordenadum

January 13th, 2016, 9:54 pm

You want want's called a sandbox.

First, anything that can write to the C disk can get to the OS and other critical files which would cross-infect the normal windows session.

Second, a user account (without admin privileges) is a bit like what you seek but its not the strong sandbox that you need.

Two solutions:

1. Virtualization would let you run multiple OSes but would require you to install software. (Can you boot from a USB drive?)

2. What about remote screen sharing with an off-site PC that can do all the things you seek?
 
User avatar
rmax
Posts: 5901
Joined: December 8th, 2005, 9:31 am

Habemus codex et ordenadum

January 14th, 2016, 9:11 am

The way forward is VM IMHO. I am tempted to run my next laptop purely as a VM.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Habemus codex et ordenadum

January 14th, 2016, 1:29 pm

Quote

Originally posted by: rmax
The way forward is VM IMHO. I am tempted to run my next laptop purely as a VM.
Indeed!

And the way forward for malware and surveillance is code that infects the hypervisor.

What I look forward to is distributed computing fabrics in which any process might use the surplus resources of the broader system (e.g., underutilized CPU, RAM, bandwidth, storage, etc.) to perform additional analyses. A process would thus have at least two types (perhaps a spectrum) of resource allocations: protected allocations (resource quantities that may not be usurped by other processes) and unprotected allocations (resources that may be suspended or usurped). To a first approximation, today's OSes seem to only offer nominally protected allocations (with difficulty!)
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

January 14th, 2018, 9:45 am

 
User avatar
Cuchulainn
Posts: 55953
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: Habemus codex et ordenadum

January 14th, 2018, 1:03 pm

tagoma wrote:

Are they 'real solids'?
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

February 9th, 2018, 8:50 am

BTW guys did you have a chance to look into observablehq?
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

February 20th, 2018, 8:15 pm

Jupyterlab anyone?
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

February 26th, 2018, 9:39 pm

tagoma wrote:
Jupyterlab anyone?

How I convinced 4A to move to Jupyterlab
Image
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Habemus codex et ordenadum

February 26th, 2018, 10:22 pm

tagoma wrote:
tagoma wrote:
Jupyterlab anyone?

How I convinced 4A to move to Jupyterlab
Image

I'll switch tomorrow, couldn't risk a failed upgrade at the  time you posted it.
I bet "Bokeh" got T4A's attention?
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Habemus codex et ordenadum

February 26th, 2018, 10:36 pm

It did notice bokeh!

The number of new environments/systems for doing computation + visualization is both wonderful and horrifying at the same time. It feels like one could spend 100% of the day trying out and learning all these systems and never getting an iota of work done.
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

March 27th, 2018, 9:35 am

C++ current date/time the crazy way


std::string foo::getTimeStamp()
{
    static const char *sqlStmt =
            "SELECT strftime( '%Y-%m-%dT%H:%M:%fZ', CURRENT_TIMESTAMP );";

    sqlite3 *db = 0;
    int sqliteRC = SQLITE_OK;
    char *sqlErr = 0;

    // Well we should return something that can be used, so picked an
    // arbitrary date, which I believe is the time of the first armistice
    // for the First World War
    std::string rval = "1918-11-11T11:11:00.000Z";

    sqliteRC = sqlite3_open( ":memory:", &db );
    if( sqliteRC != SQLITE_OK )
    {
        LOG( Log::Warn ) << "Failed to open sqlite memory DB, with error ["
                          << sqlite3_errmsg( db ) << "]";
        return rval;
    }

    sqliteRC = sqlite3_exec( db, sqlStmt, &::populate, (void*) &rval, &sqlErr );
    if( sqliteRC != SQLITE_OK )
    {
        LOG( Log::Warn )
            << "Failed to gather current time stamp"
            << " from sqlite memory DB with error [" << sqlErr << "]";
        sqlite3_free( sqlErr );
    }

    sqliteRC = sqlite3_close( db );
    if( sqliteRC != SQLITE_OK )
    {
        // We may leak some memory if this happens
        LOG( Log::Warn )
            << "Failed to close sqlite memory DB with error ["
            << sqlite3_errmsg( db ) << "]";
    }
    db = 0;

    return rval;
}
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Habemus codex et ordenadum

March 28th, 2018, 7:02 am

tagoma wrote:
C++ current date/time the crazy way


std::string foo::getTimeStamp()
{
    static const char *sqlStmt =
            "SELECT strftime( '%Y-%m-%dT%H:%M:%fZ', CURRENT_TIMESTAMP );";

    sqlite3 *db = 0;
    int sqliteRC = SQLITE_OK;
    char *sqlErr = 0;

    // Well we should return something that can be used, so picked an
    // arbitrary date, which I believe is the time of the first armistice
    // for the First World War
    std::string rval = "1918-11-11T11:11:00.000Z";

    sqliteRC = sqlite3_open( ":memory:", &db );
    if( sqliteRC != SQLITE_OK )
    {
        LOG( Log::Warn ) << "Failed to open sqlite memory DB, with error ["
                          << sqlite3_errmsg( db ) << "]";
        return rval;
    }

    sqliteRC = sqlite3_exec( db, sqlStmt, &::populate, (void*) &rval, &sqlErr );
    if( sqliteRC != SQLITE_OK )
    {
        LOG( Log::Warn )
            << "Failed to gather current time stamp"
            << " from sqlite memory DB with error [" << sqlErr << "]";
        sqlite3_free( sqlErr );
    }

    sqliteRC = sqlite3_close( db );
    if( sqliteRC != SQLITE_OK )
    {
        // We may leak some memory if this happens
        LOG( Log::Warn )
            << "Failed to close sqlite memory DB with error ["
            << sqlite3_errmsg( db ) << "]";
    }
    db = 0;

    return rval;
}


Hee $(lookup(WILMOTT_FORUM_NAME))$, that a very entertaining WTF site!
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

April 2nd, 2018, 9:40 pm

A basic question about URLs

Say I'm looking at this webpage https://apps.fas.usda.gov/psdonline/app/index.html#/app/downloads

Image

The URL to the file psd_alldata_csv.zip is https://apps.fas.usda.gov/psdonline/downloads/psd_alldata_csv.zip. It is permanent in the sense that file is updated every month while URL remains unchanged.

Now say (imagine) the information Last Updated is not available on the webpage.

Is there any metadata attached to the URL that would e.g. tell the date the data attached to have been modified? In fact, I would like to download the file only if the date that URL would get > some reference date. What query to the server to get that metadata, then?  Merci!

(My question follows a talk I overheard once. One guy was explaining that he avoided useless file downloads thanks to a priori "analysis" of the URL)
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Habemus codex et ordenadum

April 2nd, 2018, 10:37 pm

Tricky! In theory, an URL is like a directory path and file name and that's it. In practice, many sites use the URL to encode a command to a back-end database or more complex server process to retrieve and form the requested material. Sometimes the extra elements in the URL encode interesting things such as the file's date.

Unfortunately, in this example, the URL is simple and does not include any command-like elements that might encode a date.

But you might try using the HEAD method of the HTTP protocol to request the file's metadata without downloading the entire file (https://en.wikipedia.org/wiki/Hypertext ... st_methods). Or you could use the "If-Modified-Since" request header field in your request which should return just a "304 Not Modified" if the file has not changed (https://en.wikipedia.org/wiki/List_of_HTTP_headers). Whether either of those methods works will depend on how the website is built and whether the server's software is good.
 
User avatar
tagoma
Topic Author
Posts: 18033
Joined: February 21st, 2010, 12:58 pm

Re: Habemus codex et ordenadum

April 3rd, 2018, 5:23 pm

Traden4Alpha wrote:
Tricky!  In theory, an URL is like a directory path and file name and that's it.  In practice, many sites use the URL to encode a command to a back-end database or more complex server process to retrieve and form the requested material.  Sometimes the extra elements in the URL encode interesting things such as the file's date.

Unfortunately, in this example, the URL is simple and does not include any command-like elements that might encode a date.  

But you might try using the HEAD method of the HTTP protocol to request the file's metadata without downloading the entire  file (https://en.wikipedia.org/wiki/Hypertext ... st_methods).  Or you could use the "If-Modified-Since" request header field in your request which should return just a "304 Not Modified" if the file has not changed (https://en.wikipedia.org/wiki/List_of_HTTP_headers).  Whether either of those methods works will depend on how the website is built and whether the server's software is good.

Thank you for this!
The python package requests gets me what I want I think (and with only very few lines).

import requests
url = 'https://apps.fas.usda.gov/psdonline/downloads/psd_alldata_csv.zip'
r = requests.head(url)
print('{}'.format(r.headers['Last-modified']))
# Thu, 08 Mar 2018 17:04:30 GMT


The time actually is valuable. The file was available for download on the USDA/FAS website 4'30'' after the official time of data release. This is $\infty$ even for us in the world of discretionary trading!
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...