3/21/2021

Set up Jupyter in PyCharm

In 2020, PyCharm started supporting Jupitor notebook in PyCharm. After trying, it is pretty handy as it has auto-completion feature and my own plugins in PyCharm. Most importantly,  I can still use IdeaVim and my hotkeys in PyCharm.  

In this post, I am going to show I do it and demo a small stock analysis project.

Here is how I set up:

Step1: Open PyCharm and select "New Project" -> Pure Python -> Select project location and enviroment.

In my case, I chose Pipenv as I got used to it.


Step2: Install dependencies

Go to the project root and create a requirements.txt, then paste the following dependencies to requirements.txt

notebook
jupyterlab
numpy
pandas
pandas-datareader
matplotlib
pipenv install -r requirements.txt
Now, it should start installing the above dependencies.


Step3: Create ipynb file and start coding

Create an ipynb file, called StockAnalysis. ipynb, then paste the below snippet of code to it.

#%%

from datetime import datetime, timedelta

import pandas_datareader.data as web
import matplotlib.pyplot as plt

%matplotlib inline


PAST_DAYS = 365
symbol = 'AAPL'
stock_data_src = 'yahoo'

end = datetime.today()
start = end - timedelta(days=PAST_DAYS)
stock = web.DataReader(symbol, stock_data_src, start, end)
stock['MA20'] = stock['Close'].rolling(20).mean()

#%%

stock['Open'].plot()
stock['MA20'].plot(label='MV 20')
plt.legend()

Step4: Run it

With Jupyter feature, you can execute cell code(Check the screenshot). Now, you should be able to enjoy coding and doing analysis powered by Jupyter.



4/19/2020

Set up Alfred workflow for markdown edit/preview

 I've play around Alfred for a week. I found sometimes I wanted to copy markdown or some texts from somewhere, then show a preview of markdown through Alfred.

I came up an idea that maybe I can integrate Alfred with a markdown preview website to resolve this problem. Here is a result:




My Alfred workflow download for this : https://www.dropbox.com/s/jj3cofs9rczvq0j/MarkdownPreview.alfredworkflow?dl=1


11/09/2019

Deploy React app to AWS S3 in 5 mins

This post is just to show how to deploy React app to AWS S3, it does not include setting up CloudFront(CDN) and Route 53(Domain name). After going through the below steps, you should be able to deploy your react app to s3 quickly and easily. At the end of the post, there is a handy command that you can redeploy your static website next time. Here, I try to use as many screenshots as possible to show you how it works.

Step 1: Build and bundle your app

In your React app which is built with FB's create-react-app, it has a command to bundle the app out of the box:
npm run build

After running this command, it will generate a "build/" folder. Then, our goal is to upload this to S3.


Step2: Set up S3 bucket

Before doing this, please make sure you have an AWS account. If you already had an AWS account, then visit AWS S3 after entering the console: https://console.aws.amazon.com.

Click "Create bucket" button
Specify the bucket name whatever you want. Here I specified "myreactdemo" to the bucket name

Set up Block public access setting which can be changed afterward if you want. You can keep clicking next button.

Then, you will see the bucket is created
Go inside the newly created bucket, upload all contents under the "build" folder to this S3 bucket

Click Properties tab and enable static website hosting(Properties/Static website hosting). Also, specify index.html to index document and Error document fields. At the same time, copy the endpoint which will be the URL of your website.


However, after setting up this, it still demands permissions setting. Otherwise, you will see 403 error if you open the URL.


Let's fix this 403 error by changing the policy. Copy the below snippet of code to Bucket policy editor, but remember to change to your bucket name instead of mime(myreactdemo)


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[YOUR S3 BUCKET NAME]/*"
        }
    ]
}

After done with the permissions, you should be able to see PUBLIC access for your S3 bucket, then you can try to access the URL again. It should work fine.




Step 3: Redeploy

The above steps require uploading manually, you can consider using aws cli or s3cmd to make it be automatic. For instance, I use aws cli to deploy this to S3 without visiting aws console:

npm run build && aws s3 ls && aws s3 sync build/ s3://[BucketName]

Besides, you can consider hooking this up to your CI/CD like AWS CodePipeline

4/21/2019

Install and set up Ranger(Vim based file manager) on mac

Simply speaking, Ranger is a vim file manager that allows you to use the terminal to browse and modify files. This article will also include how to integrate fzf and autojmp in Ranger.

Installation

Open Terminal and install it by homebrew.
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
$ brew install ranger
$ ranger --copy-config=all

After installing, type  $ ranger to play around.


Navigation

Ranger uses many of the same keybindings as "vim". The way of navigation is similar to VIM. Below are some basic hotkeys for your reference.

j = Move down
k = Move up
h = Move to parent directory
l = Go to the next level
gg = Go to the top of the list
G = Go to the bottom of the list
tab + n = Create a new tab
tab = Switch between differernt tabs
For more hotkey information, you can reference here

Basic file operation

D = delete files
i = Display file (useful if you'd like to view a text file in a pager instead of editing it)
l or E = Open file (opens file in default file-handler)
r = Open file with… (allows you to choose program to use)
o = Change sort order (follow by character in menu selection)
z = Change settings (commonly used toggle settings)
zh = View hidden files
<space> = Select current file
t = Tag file (you can perform actions on tagged files)
cw = Rename current file
/ = Search for files
n = Jump to next match
N = Jump to previous match
yy = Yank (copy) file
dd = Mark file for cut operation
<delete> = Delete selected file

As a side note, if you would like to avoid deleting files or folder permanently by mistake, you can consider having trash-cli.
$ npm install --global trash-cli # install trash-cli

Hook up and override delete functionality in Ranger.  Map 'DD' key to delete to Trash by adding this to your ~/.config/ranger/rc.conf

$ vim ~/.config/ranger/rc.conf
Add the below line to rc.conf
map DD shell trash %s


Preview images in Ranger

Add the below two lines to rc.conf
set preview_images true
set preview_images_method iterm2

Screenshot

Integration with fzf

$ vim ~/.config/ranger/commands.py
from ranger.api.commands import Command

# https://github.com/ranger/ranger/wiki/Integrating-File-Search-with-fzf
# Now, simply bind this function to a key, by adding this to your ~/.config/ranger/rc.conf: map <C-f> fzf_select
class fzf_select(Command):
    """
    :fzf_select

    Find a file using fzf.

    With a prefix argument select only directories.

    See: https://github.com/junegunn/fzf
    """
    def execute(self):
        import subprocess
        if self.quantifier:
            # match only directories
            command="find -L . \( -path '*/\.*' -o -fstype 'dev' -o -fstype 'proc' \) -prune \
            -o -type d -print 2> /dev/null | sed 1d | cut -b3- | fzf +m"
        else:
            # match files and directories
            command="find -L . \( -path '*/\.*' -o -fstype 'dev' -o -fstype 'proc' \) -prune \
            -o -print 2> /dev/null | sed 1d | cut -b3- | fzf +m"
        fzf = self.fm.execute_command(command, stdout=subprocess.PIPE)
        stdout, stderr = fzf.communicate()
        if fzf.returncode == 0:
            fzf_file = os.path.abspath(stdout.decode('utf-8').rstrip('\n'))
            if os.path.isdir(fzf_file):
                self.fm.cd(fzf_file)
            else:
                self.fm.select_file(fzf_file)

Then, paste the below snippet to map the below commands to hotkeys. 
$ vim ~/.config/ranger/rc.conf
map cf fzf_select
map cl fzf_locate
Now, try to open Ranger, and type `cf`. It will show fzf search console


Integration with autojump

With integration with autojump, it makes you faster to navigate your filesystem.
First, create rc.conf in ~/.config/ranger
 $ vim ~/.config/range/rc.conf
Paste the key mapping between the below plugin and hotkey.
map cj console j%space

Second, create a plugin for that. After finishing this step, you can open ranger again, then type 'cj' to execute the autojump.
$ mkdir -p ~/.config/ranger/plugins
$ vim ~/.config/ranger/plugins/autojump.py
Paste the below code to autojump.py. You can reference my settings and code here.
import ranger.api
import subprocess
from ranger.api.commands import *

HOOK_INIT_OLD = ranger.api.hook_init


def hook_init(fm):
    def update_autojump(signal):
        subprocess.Popen(["autojump", "--add", signal.new.path])

    fm.signal_bind('cd', update_autojump)
    HOOK_INIT_OLD(fm)


ranger.api.hook_init = hook_init


class j(Command):
    """:j
    Uses autojump to set the current directory.
    """

    def execute(self):
        directory = subprocess.check_output(["autojump", self.arg(1)])
        directory = directory.decode("utf-8", "ignore")
        directory = directory.rstrip('\n')
        self.fm.execute_console("cd " + directory)

4/15/2019

Set up Visual Studio Code for python in few minutes

A few days ago, I began working on a python project again. It has been over a year since I wrote python. Back in the days, I chose VIM and Pycharm for python when it comes to IDE. However, I started shifting some jobs or projects to visual code recently. Also, I got used to Visual Studio Code a little bit. Therefore, I spent a day to search and understand how many extensions are required to make Python development in Visual Studio Code a better experience. Here, I will list useful extensions and python packages I installed.

Python packages


autopep8 automatically formats Python code to conform to the PEP 8 style guide:

pylint is a static code analysis tool which looks for programming errors:

If you choose pytest for your unit test, you have to install those:

Visual Code Extensions


1. Python Microsoft

Download: https://marketplace.visualstudio.com/items?itemName=ms-python.python
An extension with rich support for the Python and it contains linting, debugging, code navigation, code formatting, refactoring, unit tests, snippets, and so and so. This is a basic and necessary extension!

2. Python autopep8

With this, it can allow you to do auto-formatting in your Visual Studio Code.

3. py-coverage-view

The extension highlights test coverage in Visual Studio Code. It helps me to understand which line is not tested without looking at the console output.

4. Python Indent

I just installed because it looked neat.

5. GitLens — Git supercharged

GitLens helps you quickly find out whom, why, and when a line or code block was changed.

6. Trailing Spaces

It helps you remove annoying trailing spaces..

7. autoDocstring

This one makes you write python document easier.

8. Importmagic

Download: https://marketplace.visualstudio.com/items?itemName=brainfit.vscode-importmagic
It claims it can help me to import module automatically, but it seems it cannot work on my VSCode for unknown reasons.

9. Spell check

Download: https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker
Needless to say, this extension is to check your spelling in your code or README.

10 Material-icon-them

Download: https://marketplace.visualstudio.com/items?itemName=PKief.material-icon-theme
This is an extension to beautify your folder and file with icon-based Material design.

11 Bracket Pair Colorizer

Download: https://marketplace.visualstudio.com/items?itemName=CoenraadS.bracket-pair-colorizer
It makes the code easier to identify matching brackets like ( [ { } ] ).

12 Python Test Explorer

Download: https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
This extension allows you to run your Python Unittest or Pytest tests with the Test Explorer UI.

13. VIM

Download: https://marketplace.visualstudio.com/items?itemName=vscodevim.vim
For Vim users, it is a Vim emulation for Visual Studio Code
As a side note, if you are a mac user and you want to press vim navigation key in mac to move the cursor, you have to type this in your terminal 
defaults write com.microsoft.VSCode ApplePressAndHoldEnabled -bool false

14. GlobalConfig

After installing certain extensions, you might need to configure the settings for each project. GlobalConfig offers a solution to this. Therefore, you can create a global setting under ~/.vscode/settings.json which has the default settings. Here is an example

15. Sync your VSCode settings

Once you installed so many extensions in your VSCode, there is another extension to sync your settings across machines. For more information, you can follow this page: https://itnext.io/settings-sync-with-vs-code-c3d4f126989

Note

As a side note, VSCode keeps track of each folder if they were changed. As a result, it takes up lot of memory and opens lots of file descriptors, leading to horrible performance on your computer. If you want to exclude certain folders or files, you can reference this post.
https://stackoverflow.com/questions/33258543/how-can-i-exclude-a-directory-from-visual-studio-code-explore-tab



4/07/2018

Install PlantUML environment on Mac

PlantUML is a tool to draw UML. It is a tool that you can write a text description to draw your UML diagram.

If you want to use it on your local mac desktop, you can use Homebrew to install that. Before installing PlantUML, make sure that you have installed Java and Graphviz.

Make sure that you have installed Java:
 $ java -version

Install Graphviz

$ brew install graphviz

Now, install PlantUML:
$ brew install plantuml

Let's try a sample.

$ wget https://gist.githubusercontent.com/QuantumGhost/0955a45383a0b6c0bc24f9654b3cb561/raw/08e6eb965a5604a8ba89a586707708a9a9b98c1a/example.puml
The below command will dump to png file
$ plantuml example.puml
The below command will diaplay the latest result
$ plantuml -gui example.puml 

As a side note, you can integrate it with Visual Studio code or sublime editor. If you are using Visual Studio code, I would recommend PlantUML extension which is able to show a live preview.

A sample for planting sequence diagram:
http://plantuml.com/sequence-diagram


7/26/2017

Install Chewing on Ubuntu 16.04

Install ibus-chewing

sudo apt-get purge hime hime-chewing
sudo apt-get install ibus-chewing 
im-config -n ibus 
ibus-setup 
ibus restart


Add Input Method

Go to  System settings -> Text Entry, add 'Chinese(Chewing)'



7/20/2017

[Deep Learning] Batch Normalization note

Batch Normalization (BN)
• Normalizing input (LeCun et al 1998 “Efficient Backprop”)
• BN: normalizing each layer, for each mini-batch
• Greatly accelerate training
• Less sensitive to initialization
• Improve regularization

S. Ioffe & C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML 2015

'Deep Residual Learning for image recognition', presents that do batch normalization before active function will be better.


7/19/2017

Vagrant command lines

Install Vagrant from official website

  If you use apt or yum to download Vagrant, the version might be too old to be compatible to your virtualbox or vmware. Therefore, please download it from the official website .


Download VirtualBox from website

  Please download the suitable version of VirtualBox.


Common and useful commands:

Download and init vagrant box
$ vagrant init <Vargant box>
E.g : Download vagrant box which has Kubernetes.
vagrant init flixtech/kubernetes

Note: The box file will be stored in ~/.vagrant.d/boxes/
Creates and configures guest machines according to your Vagrantfile.
$ vargant up

Bring the machine up with the given provider. By default, this is "virtualbox".
$ vargant up --provider x

E.g : Change the provider to vmware
$ vagrant up --provider vmware

Check the current machine states
$ vagrant status

List all of base boxes for Vagrant
$ vagrant box list

Shut the current machine
$ vagrant halt

Stops the running machine and destroys all resources that were created 
$ vagrant destroy

SSH into a running Vagrant machine
$ vagrant ssh

JFrog setup gradle artifacts

Run JFrog

Create a directory which stores JFrog's persistent data, then run JFrog using Docker. We change the PORT to 8085 instead of using 8081 because 8081 port was occupied by my Jenkins server.
$ mkdir -p jfrog_data 
$ docker run --name jfrogartifactory -d -v jfrog_data:/var/opt/jfrog/artifactory -p 8085:8081 docker.bintray.io/jfrog/artifactory-oss
I created a makefile for running and stopping the container and backup data.
init:
 mkdir -p jfrog_data
run:
 docker run --name jfrogartifactory -d -v `pwd`/jfrog_data:/var/opt/jfrog/artifactory -p 8085:8081 docker.bintray.io/jfrog/artifactory-oss
stop:
 docker stop jfrogartifactory;docker rm jfrogartifactory
clean:
 rm -rf jfrog_data
 docker rmi -f docker.bintray.io/jfrog/artifactory-oss
backup:
 zip -r jfrog_data.zip jfrog_data
After running JFrog, you can check it by entering http://localhost:8085/

Create gradle and generic repositories


Create a group, permission, account




Log out of admin account

Publishing your library using Gradle

Step 1: Add org.jfrog.buildinfo:build-info-extractor-gradle:4.0.1 dependency to the project gradle
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.2.2'
        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.0.1"
    }
}
Also, add the variables to gradle.properities to let module use
artifactory_user=XXXX
artifactory_password=YYYY
artifactory_contextUrl=http://10.70.70.40:8085/artifactory

Step 2: Append the below snippet code to then end of your module's gradle file 

apply plugin: 'com.jfrog.artifactory'
apply plugin: 'maven-publish' def libraryGroupId = 'AA.BB.CC' def libraryArtifactId = 'Test' def libraryVersion = '1.0.0' publishing { publications { aar(MavenPublication) { groupId libraryGroupId version libraryVersion artifactId libraryArtifactId artifact("$buildDir/outputs/aar/${artifactId}-release.aar") } } } artifactory { contextUrl = "${artifactory_contextUrl}" publish { repository { repoKey = 'gradle-release-local' username = "${artifactory_user}" password = "${artifactory_password}" } defaults { publications('aar') publishArtifacts = true properties = ['qa.level': 'basic', 'q.os': 'android', 'dev.team': 'core'] publishPom = true } } }

Go to command line to publish your module to your private JFrog artifactory
$ ./gradlew assembleRelease artifactoryPublish
Then, your artifacts will be located at http://localhost:8085/artifactory/webapp/builds/

Import the library in other projects

Step 1: Add repository dependency

    repositories {
        jcenter()
        maven {
            url "http:/[IP]:8085/artifactory/gradle-release-local"
        }
    }
Step 2: Import the library in your module and compile it
compile "com.pakcage:module:version"
It's done!


Upload other artifact using JFrog

Jfrog with free version supports that you can create a generic version and upload anything

Then, you can use curl command to push your artifact to that. Ex:
curl -u<USERNAME>:<PASSWORD> -T <PATH_TO_FILE> "http://10.70.70.44:8085/artifactory/<REPO KEY>/<TARGET_FILE_PATH>


My makefile file : https://github.com/tzutalin/docker/tree/master/JFrog