Top_Banner

May 26, 2018

Auto-refresh ImportXML at predefined intervals using Time-Driven Triggers on Google Script

How to create 'Time-driven Triggers' in Google Script 

Functions that pull data from outside the spreadsheet recalculate at the following times:

      ● ImportRange: 30 minutes
      ● ImportHtml, ImportFeed, ImportData, ImportXml: 1 hour
      ● GoogleFinance: may be delayed up to 20 minutes

See more details here.


Time-driven triggers is a very helpful tool for getting periodic updates of [import] functions used in google spreadsheets. There are two ways to do it. First script uses a random number added to the target page as an URL query string.  It by passes the one hour limit by  calling a dummy URL every time at auto-recalculation of google spreadsheet. Since target values are populated directly through scripted formulas, it is tedious to manipulate a large number of function through with it.

 Lets see, how can we set-up a Time-driven triggers in script editor for import functions at a time interval as low as 'on every minute'.  It would be best to use it for upto 5-10 import function, but if you handle more , than go ahead as there is no restriction as such except script may take a little longer to update.

Make sure all you have is a list of URLs and Xpath_query  that you want to get output to values in output Cells after each auto-update.  if you need any help on [Xpath query]  generation see related article or video blog at :

Related : Use Xpath query to fetch data from a Webpage


Just copy and paste the simple code in google script editor :





function getData() {
  var queryString = Math.random();

  var Xpath_1 = "[Your Xpath_query here]";
  var importXpath_1 = '=IMPORTXML("' + '[Your URL link here]' + '?' + queryString + '","'+ Xpath_1 + '")';
  
  SpreadsheetApp.getActiveSheet().getRange('[outputCell]').setValue(importXpath_1);
}

Replace desired values at <[Your Xpath_query here]>, <[Your URL link here]> and <[outputCell]in the above mentioned code. save project and allow permissions to 'Run' the script in your gmail account.

Related : Auto-updated Equity Stocks Portfolio in Google Sheets

Utility script for periodic refresh of IMPORT formulas

Above said basic script works well, if you want to populate just a few  import functions. It not easy to manipulate a long list of import functions inside a spreadsheet. I explored the web for this excellent solution that works great. Thank you all contributors to share such a wonderful script. 


Here is the script, :

function RefreshImports() {
  var lock = LockService.getScriptLock();
  if (!lock.tryLock(5000)) return;             // Wait up to 5s for previous refresh to end.

  var id = "[YOUR SPREADSHEET ID]";
  var ss = SpreadsheetApp.openById(id);
  var sheet = ss.getSheetByName("[SHEET NAME]");
  var dataRange = sheet.getDataRange();
  var formulas = dataRange.getFormulas();
  var content = "";
  var now = new Date();
  var time = now.getTime();
  var re = /.*[^a-z0-9]import(?:xml|data|feed|html|range)\(.*/gi;
  var re2 = /((\?|&)(update=[0-9]*))/gi;
  var re3 = /(",)/gi;

  for (var row=0; row<formulas.length; row++) {
    for (var col=0; col<formulas[0].length; col++) {
      content = formulas[row][col];
      if (content != "") {
        var match = content.search(re);
        if (match !== -1 ) {
          // import function is used in this cell
          var updatedContent = content.toString().replace(re2,"$2update=" + time);
          if (updatedContent == content) {
            // No querystring exists yet in url
            updatedContent = content.toString().replace(re3,"?update=" + time + "$1");
          }
          // Update url in formula with querystring param
          sheet.getRange(row+1, col+1).setFormula(updatedContent);
        }
      }
    }
  }

  // Done refresh; release the lock.
  lock.releaseLock();

  // Show last updated time on sheet somewhere
  sheet.getRange(7,2).setValue("Rates were last updated at " + now.toLocaleTimeString())
}




Resources :

1. Periodically refresh importxml() spreadsheet function

May 23, 2018

Xpath query to fetch data from a Webpage

Don't need to be an coding expert to use Xpath query . . .!!
                to fetch data from any web page.

Internet as a source of information has penetrated too deep in our lives, so much that we always look at google for solutions to any of the issues whether related to personal or professional life. Web is flooded with plenty of resources on any topic related to human being. However, main content on any web page is either Static or Dynamic. Structured data on Static pages as are rarely modified after it published on web. Therefore, no need to get any dynamic update for such pages, use simple tools to grab the data and work forever.

However, Dynamic pages are actively maintained and update too frequently with real time data that is published in a pre-defined defined and standard page format.  Such pages and content includes :

          ¤  Temperature or Weather forecasts
          ¤  Gold or Currency rates
          ¤  Real time Stocks and Commodity Price quotes
          ¤  Daily Net Asset Values for Mutual funds.
          ¤  Social Media Profile performance such as Likes, page views, Followers, Retweets  etc .
          
I started it all to get 'Daily Net Asset Values'  for Schemes in my Mutual funds Portfolio.

Although it is possible to get Live quotes (Delayed) for NSE/BSE stock through a simple and built-in function in google sheets, but it doesn't work for mutual funds as of now. 

See it here : [ Auto-updated Equity Stocks Portfolio in Google Sheets ]

Equity stock Portfolio works well for any stock supported by Google Finance. But it is not possible to get any data other than a list of pre-defined attributes.  Another issue as state earlier, It does not support mutual funds. Therefore, after studying a lot of resources, I figured out a pre-defined function in google sheet that is an excellent way to get daily NAVs directly from AMFI official website. 

See it here :  [Auto-update ] Mutual Funds NAVs Daily from AMFI website . . . 

AMFI provides NAVs in text format. So, above method did not Fully meet the desired intent to fetch additional fund performance data for better comparison of schemes. It would be great to have past returns, risk measures, expense ratios, assets and exit loads if any. I had to go into a little deeper concepts and not so common functions in google sheets or Libre office calc. In the meantime, I need to mention it in very strong words that I never took Google sheets so seriously despite its huge capabilities and such a strong built-in functions, nothing less than any of popular spreadsheet applications in market today.

Whole stuff is based on just one pre-defined function i.e importxml
Syntax  :  [ =importxml ( " URL ", " Xpath_query ") ]

However, things are not so simple as it seems. Xpath_query  is the critical part in the above syntax. Let see it details.

What is Xpath ?
Xpath stands for XML Path Language.
Most of us have a very good understanding of a Folder Tree or  Root Directory Structure in any of conventional file system such as windows explorer. Xpath uses same analogy of Parent-Child relationships and folder path structure to navigate to a specific file or sub-folders. 

Have a look at a basic XML document structure,


XML documents too has a "path like" syntax to identify and navigate nodes.  
[elements] are the building blocks of any xml document. These so arranged in an hierarchy that includes multiple nodes as well as sub-elements, to create a structured data-set on a landing page. It is very easy to notice common elements such as <div><span> . As you explore that complete code, some other common elements are <table>, <tbody>, <tr> , <td> , <a>, <li>, <ul> etc. 

[attributes] are like <tag>  or custom names assigned to independent nodes, It helps to directly jump over a specific node or element by-passing all elements from the beginning to the target element in a xml document. It is same like calling someone by name.  These two attributes are  too common <class>, <id> than others.

Syntax :         "// <#element> [ @ <#attribute> = '<#name>' ]" 

Highlighted part is termed as Predicate. It is used to find a specific node or node that contains a specific value. for example, Xpath_query to fetch  "42" in above image :

                       "// div [ @class = 'today_nowcard-temp']/span"

Here, <span> is the target element that contains the desired value. Predicate part would direct look for the class value equal to defined <#name>
if you closely look at above syntax,  "//. . . ./ . . " doesn't it look like a folder path ??


Simple Method to get a ready made Xpath query?
It requires some practice, little patience and a number of trials to master the coding  for xpath query using above Syntax method. But once you do it,  It feels like a pro in Google sheets than ever before. 

Cheers!! We have another simple way to get Xpath query
if you are not comfortable enough to generate a manual syntax at you own. 



Have you even noticed,  an option inspect ] given at extreme  bottom, if you right click at "selected data" on a page in google chrome. It will split the screen into two parts.  Bottom part would have the selected data highlighted in xml document. Here, It is required to inspect the highlighted element structure to generate a Xpath query.

        Or

Proceed to next image for the simple version,



Right click on the Highlighted target element in the bottom window, Select 'Copy XPath' as shown in image above.



Lets move on how to use this Xpath query with two example videos.

● Auto-Update Portfolios with Stock Quotes and NAVs ● 



● Fetch Latest Temperature using Xpath query in Google Sheets ● 





        Resources :              

May 10, 2018

Sync Multiple Google Drive accounts on Desktop

Transfer or Move folders from Primary account to any other Google account.

Have you exhausted 15GB Free storage in Primary Google account ???
.
.
Do you want to recover 'free-space' and also not to delete any of your files and folders ?

It is possible to do it. 


Latest updates in 'Backup and Sync from Google' has added a very helpful feature to concurrently Sync upto 3 google drive accounts on desktop. Before that it was 'just a work around' to transfer data between two google accounts.
How it was done earlier ?? See it Here.

[Desktop Sync for Multiple Google Drive accounts.]
Let us see, how to do it.



['Add new account' ] feature enables  Sync of multiple drive account with Local desktop. Therefore, Data management is too simple now with [Cut, Copy and Paste] functions in windows explorer. Users can easily  Cut all folders to transfer whole drive to another account.

May 6, 2018

Daily updates for Net Assets Values (NAVs) of mutual fund schemes directly from AMFI website . .

Mutual Funds Portfolio using [=Importdata] function in Google Sheets


Download link :
AMFI_Auto-updated Mutual Funds Portfolio Spreadsheet | Google Sheet

To get an automatic update for daily NAVs directly imported from AMFI data, follow the steps given in video tutorial. 
  • Google Sheet Functions 
=IMPORTDATA("https://www.amfiindia.com/spages/NAVAll.txt")


=SPLIT(A1,";",TRUE,FALSE)


=VLOOKUP(B2,amfi_import!D:J,3, FALSE)

  • Also, It can be combined in same worksheet as : 

      May 5, 2018

      Auto-updated Equity Stocks Portfolio in Google Sheets

      Although I also tried to search the web to get a 'ready-to-use'  tool to manage an already existing 'Static' investment portfolio that is being managed in spreadsheets, but it ended-up with a discrete set of resources with no exact solution. Among several options, It is interesting to see online portfolios services provides by  some well known financial portals, that just meet the basic requirements if you don't have an online trading account. They have a almost similar dashboard showing standard parameters including Holdings statement, Actual cost, Current Value, LTP, %change and Profit/Loss status etc. However, It is an integral feature at most online trading platforms in India, with Buy or Sell security as additional function at  same dashboard.  It doesn't make a sense to have an just the same portfolio somewhere else as well.

             Other alternatives includes complex macros-enabled excel sheet, built-in VBA scripts to import/fetch live quotes data from leading  financial portals. However, They offer a limited customisation as well as frequent updates at source portal breaks the link with spreadsheet. It requires a latest updated version of script to adapt changes. Average user doesn't have skills to edit these scripts and have to wait for the updated version.

      To sum it upI couldn't find a better alternative to 'Static' portfolios spreadsheets. There is no doubts that spreadsheets are the best tool to manage tabular data when it comes to ease of use, flexibility and end-user customisation. But It is too boring, cumbersome to update every single bit of price quotes and financials, every time before doing an assessment and portfolio analysis to make a fair investment decision.

      Using google sheets,  I have finally migrated to a better, dynamic and interactive investment portfolio that features a simple, yet effective way to get live quotes for both NSE/BSE stocks. 

      Why to make a portfolio with Google Sheets ??

      It is the simplest way!!!

      Complex scripts are required to fetch live quotes in excel and other spreadsheet programs. Google sheets has a unique function [=googlefinanceto with a list of pre-defined attribute parameters to directly get the live quotes from finance.google.com.  Google scripts editor provides an additional interface to create custom functions at advanced levels.

      Anybody with basic excel skills can easily customise or even start from scratch for one's own portfolio !!

      Pre-Requisites : 
      • Create a new blank google sheet.
      • Capture Security Code or NSE symbols.
      Security code or NSE stock symbol is the key is must have requirement to get live quotes.  
      Here are the ways, how can you get it !!


      However, if you need bulk stocks in a specific index, it is better to directly copy from  NSE or BSE website.

      NSE stocks :
      https://www.nseindia.com/live_market/dynaContent/live_watch/equities_stock_watch.htm



      BSE stocks :



      Auto-update for mutual funds NAVs is a bit more tricky as google finance is not working for it. So, this blog is just limited to stocks portfolio.
      • Create column structure for equity portfolio 
      Sample portfolio structure is shown below, however it is relatively easy to customise it for your requirements.


      • Syntax for [google finance] function to fetch Live quotes . . .
      After having security codes and symbols, now its time to use them to fetch live quotes. I am using [TCS] for NSE quotes in below examples.



        Stock Name 

        =googlefinance("NSE:TCS", "Name")


        Trade Time 


        =googlefinance("NSE:TCS", "tradetime")


        Last Trade Price 

        =googlefinance("NSE:TCS", "price")


        % Change 

        =googlefinance("NSE:TCS", "changepct")

        52w High 

        =googlefinance("NSE:TCS", "high52")

        52w Low

        =googlefinance("NSE:TCS", "low52")

        Market Cap.

        =(googlefinance("NSE:TCS", "marketcap"))/10^7

        Earning per Share

        =googlefinance("NSE:TCS", "eps")

      if you are interested to see a complete list of attributes, must see its Support page .


      • Equity portfolio has built-in script to add  security  step-by-step. 
      • However, first 80 rows are already populated with formulae. 
      • Just copy-paste the values in exchange [Column 'A'] and security code in  [[Column 'B']. That's it !!