Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Tuesday, March 24, 2009

erlang和python交互

最近开发 Erlang ,对其字符串处理能力无言至极,于是决定把它和python联合起来,打造一个强力的分布式系统,等将来需要系统级开发时,我再把 C++/C组合进来.

首先参考了 Erlang 官方文档和 http://blog.developers.api.sina.com.cn/?tag=erlang 以及 http://kazmier.net/computer/port-howto/ .

研读了将近24个小时, 才终于完全把问题解决. 开始 python 一直是broken pipe.后来才知道是管道中断导致的问题.

接着我排除了python的问题, 在erlang上面发现了Timeout, 感谢litaocheng的提醒,翻看了文档,把gen_server:call 的超时控制为infinity, 这样完全排除了超时的错误.

后来发现执行combin()之后,系统一直在等待. 我重新参考了下前面的2个成功案例,数据流的结尾应该
加上"\n"(windows为"\t\n"),并且erlang需要转换二进制,于是就是

String = list_to_binary("combine|"++_String++"\n"),

python端就成功收到了Stream.
这里combine只是我自己定的一个通信协议 合成请求|String1,String2 => string1+string2.








%%%-------------------------------------------------------------------  

%%% File    : gen_server_template.full  

%%% Designer  : free.Won <freefis@Gmail.com>  

%%% Description :  

%%%     Communication between Python and Erlang via Stdio.  

%%% Archieved :  Mar 24 2009 by  free.Wang <freefis@Gmail.com>  

%%%-------------------------------------------------------------------  

-module(comm).  

-behaviour(gen_server).  

%% API  

-export([start/0,combine/1]).  

%% gen_server callbacks  

-export([init/1, handle_call/3, handle_cast/2, handle_info/2,  

     terminate/2, code_change/3]).  

-record(state, {port}).

  

start() ->  

    gen_server:start_link({global, ?MODULE}, ?MODULE, [], []).

init([]) ->  

    process_flag(trap_exit, true),

    Port = open_port({spawn, "python -u /home/freefis/Desktop/comm.py"},[stream,{line, 1024}]),

    {ok, #state{port = Port}}.

handle_call({combine,String}, _From, #state{port = Port} = State) ->  

    port_command(Port,String),

    receive

        {Port,{data,{_Flag,Data}}} -> 

            io:format("receiving:~p~n",[Data]),

            sleep(2000),

            {reply, Data, Port}

    end.

handle_cast(_Msg, State) ->  

    {noreply, State}.  

handle_info(Info, State) ->

    {noreply,State}.

%    {stop, {port_terminated, Reason}, Port}.

terminate(_Reason, Port) ->  

    ok.

code_change(_OldVsn, State, _Extra) ->  

    {ok, State}.  

  

%%--------------------------------------------------------------------  

%%% Internal functions  

%%--------------------------------------------------------------------

sleep(T) ->

    receive

        after T ->

            true

    end.

combine(_String) ->

    String = list_to_binary("combine|"++_String++"\n"),

    gen_server:call({global,?MODULE},{combine,String},infinity).








#!/usr/bin/env python

# -*- coding:utf-8 -*-

#-------------------------------------------

# Designer  : Free.Wang

# E-mail    : freefis@Gmail.com

# Licence   : License on GPL Licence

# Archieved : Dec 27 2008

#-------------------------------------------







import sys



def handle(_string):

    if _string.startswith("combine|"):

        string = "".join( _string[8:].split(","))

    return string







"""waiting for input """

while 1:

    # Recv. Binary Stream as Standard IN

    _stream = sys.stdin.readline()

    if not _stream: break

    # Scheme, Turn into  Formal String

    inString  = _stream.strip("\r\n")

    # handle String

    outString = handle(inString)

    # send to port as Standart OUT

    sys.stdout.write("%s\n" % (outString,))

    sys.exit(0)


Saturday, March 14, 2009

python发送post保存cookie封装

近期刚写完某SNS的一个刷人气机器人(仅仅做技术测试),然后封装了这个功能。

socket = Httpsocket()
创建一个请求实例,并初始化 cookie容器

socket.connect(url,param)
请求处理post的url和post 参数 执行完后 会返回该socket

socket.openurl(url)
通过持有上一步中得到的cookie来访问接下来的网络地址。


# auther: free.Wang<freefis@gmail.com>

import urllib
import urllib2

class Httpsocket:
 """ Build for Make a full HttpRequest via POST/GET """
 def __init__(self):
# login url to recieve param , Post/Get param
# opener will be used to open Url with cookie.
# build the cookie container.
  self.cookies = urllib2.HTTPCookieProcessor()
  self.opener = urllib2.build_opener(self.cookies)
  urllib2.install_opener(self.opener)

 def connect(self,loginurl,param):
 """make a Container for Cookie"""
  self.loginurl = loginurl
  self.param = param
  self.encodeparam = urllib.urlencode(self.param)
  try:
   urllib2.urlopen(urllib2.Request(self.loginurl,self.encodeparam))
  except:
   pass

 def openurl(self,url):
 """ get Response. """
# serialize the Param
# Login to Get Cookie ,Which will be Push into self.opener.
# get request with Cookie.
  return self.opener.open(url)

Thursday, January 1, 2009

Python2.5和python3.0性能循环测试

测试环境 Ubuntu 8.04 linux 2.6.24-23-386 i686 GNU/Linux

运行环境 python2.5/python3.0

分析命令:Top

测试结果分析:
分界线上段代码的用的无限循环。然后到某个值后跳出
1,2分别为python2.5,python3.0的执行环境
分界线下段代码的用的是遍历range
3,4分别为python2.5,python3.0的执行环境

先分析Python2.5
1,3 做横向比较, 1的CPU的消耗略小于2,而内存消耗明显高于2。
原因很简单:1 中 对i做累加,所以略占用计算资源。而2则是在内存中实现一组长序列(请原谅我未曾深入python C源码学习,range实现的数据结构我不了解)显然内存空间被大量占用。

再分析python3.0
2,4做横向比较, 内存消耗相差不大。因为python3.0中的range实质是从List->Iterator.但是CPU消耗是很大的。这个我们最后总结。

纵向比较1,2和3,4, 1的各方面都优于2。而3,4分别优于CPU和内存。

总结: python3.0 对 python2.5的数据结构做了很大改进(这里我们谈实现性能)。总体方向是很成功的。问题是,细节上,需要优化的地方太多太多,就像Guido之前所说。而至于python3.0 在实现上的改进的所有内容,请参见python.org.

就目前主流的 python2.5而言。把range作为循环的容器显然是很消耗资源的行为。
所以以后就该有个好习惯,如果循环序列很大,那么就用while做为一个好习惯。

虽然我们常说空间换时间,但在这种情况下,无非舍本逐末。

#!/usr/bin/env python
#-*- coding:utf-8 -*-

#-------------------------------------------
# Designer : free.Won
# Licence : License on GPL Licence
# Archieved : Jan 1st 2009
#-------------------------------------------

import threading

class Multirun(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
i=1
while 1:
if i== 10**7:
break
print(i)
i+=1

if __name__ == '__main__':
tset = []
for i in range(10000):
t = Multirun()
tset.append(t)
for one in tset:
one.start()
one.join()
1.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17305 freefis 20 0 23152 12m 1524 S 10.3 0.7 0:03.36 python

2.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17451 freefis 20 0 29836 18m 1720 S 28.0 1.0 0:08.92 python3.0


===========================================

#!/usr/bin/python3.0
#-*- coding:utf-8 -*-

#-------------------------------------------
# Designer : free.Won
# Licence : License on GPL Licence
# Archieved : Jan 1st 2009
#-------------------------------------------

import threading

class Multirun(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
for i in range(10**7):
print(i)

if __name__ == '__main__':
tset = []
for i in range(10000):
t = Multirun()
tset.append(t)
for one in tset:
one.start()
one.join()

3
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17389 freefis 20 0 176m 166m 1524 S 9.0 8.8 0:02.76 python

4
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17494 freefis 20 0 29836 18m 1720 S 47.9 1.0 0:10.46 python3.0

Saturday, December 27, 2008

Python3.0 Urllib体验

首先推荐大家接触python3.0. 有争议的东西毕竟是好东西.
http://python.org/ftp/python/3.0/Python-3.0.tar.bz2 可下载python3.0 可以推荐大家玩玩. 如果担心和当前python环境影响没关系.解压后进入该目录.

sudo su
mkdir /opt/python3.0
./configure --prefix=/opt/python3.0 && make && make install
ln -s /opt/python3.0/bin/python3.0 /usr/bin/python3.0



python3.0出了之后.像我这样热衷爬虫的偏执狂最关心的就是它的urllib.
可以去python.org看看文档,urllib是修改比较剧烈的模块.urllib2和urllib被整合到了urllib包.相应的方法都被归到不同模块中. 在下面的代码中,urlopen, urlencode 被定义在request,parse模块.




#!/usr/bin/python3.0
# -*- coding:utf-8 -*-

#-------------------------------------------
# Designer : Free.Wang
# E-mail : freefis@Gmail.com
# Licence : License on GPL Licence
# Archieved : Dec 27 2008
#-------------------------------------------

from urllib import request,parse

class Makesocket:
""" Build for Make a full HttpRequest via POST/GET """
def __init__(self,loginurl,param):
# login url to recieve param , Post/Get param
self.loginurl = loginurl
self.param = param

def cookie(self):
"""make a Container for Cookie"""
self.cookies = request.HTTPCookieProcessor()
# opener will be used to open Url with cookie.
# build the cookie container.
self.opener = request.build_opener(self.cookies)
request.install_opener(self.opener)

def run(self):
""" get Response. """
# serialize the Param
# Login to Get Cookie ,Which will be Push into self.opener.
# get request with Cookie.
self.encodeparam = parse.urlencode(self.param)
request.urlopen(self.loginurl,self.encodeparam)
conn = self.opener.open("http://www.hellofreefis.com/profile/")
print(conn)

if __name__ == '__main__':
oginurl = "http://www.hellofreefis.com/info_oper.php?tag=signin&pageref="
param = {
'name':"freefis",
'pwd':"freefis' passwod"
}
conn = Makesocket(loginurl,param)
conn.cookie()
conn.run()


测试成功.Py3.0的改动还是很大的.但愿Python4.0出现的时候,不会再来革命.我个人角度对这次改动还是很肯定.感谢Guido把reduce函数保留到functools模块中.
希望各大公司和*nix操作系统发行者能尽快支持python3.0.

P.S:以上代码让我觉得很"悲剧".blogspot转义空格的支持不够到位.若要迁移代码的朋友,请自行缩进.

Followers